I keep looking at these neatly drawn formations of the Swedish brigade (their basic infantry tactical unit) and it looks as if the companies of shot in the center have to fire over or through the line of pikes in front of it. And with the wings of shot extending beyond the lines of pike both right and left, how did the form a continuous defensive line? How did they form a square against cavalry charges with only three lines of pike? The formation as shown does not seem to make any sense:
PPPP
SSSS
SSSS PPPP.........PPPP SSSS
SSSS
First of all, the tactical unit of the Swedish army at this time is the infantry squadron (a battalion consisting of four infantry companies of which about 2/3 were pikemen). These could be grouped into brigades, which consisted of 2-4 infantry squadrons plus light field artillery and its own reserve. The Swedish brigade was about the same size in men as a Tercio, but as opposed to a Tercio, it was NOT a tactical unit.
The Swedish infantry was grouped six ranks deep and the musketmen would double up (close to three ranks) and fire salvo fire. From the Battle of Lützen 1632, both pikemen and musketeers were deployed three ranks deep from the beginning.
The formation you have drawn is actually very effective against a Spanish Tercio.
Here's a drawing of the Battle of Lützen 1632. Note the T-shaped Swedish brigades (in the upper part of the image). They form a neat line and protect each other's flanks.
The infantry in your diagram were supposed to move in relation to each other. In purely defensive warfare, the formation would remain as shown in the image and in your diagram. The musketeers behind the front pikemen could march through the pikemen and fire, or take position on either flank to fire. Likewisde the musketeers to the rear. If an enemy pike formation reaches the front pikes, the muskeers on the lanks further back can pour fire into them. If the enemy pike formation attempt to shape its line as an U and attack the front pikemen on their flanks, the rear pikemen can attack their flanks while the rear musketeers pour fire into their backs.
It is actually very hard to attack this formation without either exposing idle men to constant musket fire or exposing your own flanks to pike and musket fire. Likewise, on the attack the pikes on the front can hold your formation while the musketmen pour fire into it and then the pikes can attack.
You must also remember that on the flanks of the musketmen on the respecive flank there would be light field artillery that can pour grapeshot down your own ranks.
These formations were not intended to form squares or even be able to make a 90 degree turn to change their frontage (not quickly anyway). That was the job of the second line, which would also send formations to plug the gap should a large one appear in the line. It was the second line that marched to form a new flank after the Saxonians fled during the Battle of Breitenfeld 1631 and then held the Imperial cavalry. Remember that this is the age of formation warfare and especially pike formations need to remain together to be effective - if you lose the pikeman to your side, you are basically unprotected, so even if there are small holes in the line, troops could not just quit their formation, stream through and reform in formation on the other side to attack.
Imagine it as legos. You have one large block (a Tercio) and several small blocks (Swedish infantry squatrons). The large block can't get through small gaps between the small blocks. The small blocks can attack the flank or even the rear of the large block while it tries to take out one of the small blocks.
Basically, the Swedish formation, by being mobile and allowing the various part of it to attack or defend against an enemy depending on how he was formed or moved made it a 'damned if you do, damned if you don't' or 'no-win' scenario.
If the Tercio attacked the pikes from the front, it would face musked crossfire supported by light artillery.
If the Tercio squeezed between the front pikes, it would face fire and then pikes from the front, and pikes and fire from the flanks.
If the Tercio tries to envelope the front pikes, it faces pikes on its flanks and fire in its rear.
The Swedish formation allows a lot of men to fire or use pikes against the enemy, while the Tercio has a lot of men whose only contributio will be adding their weight to the pike push from the rear ranks.
As for cavalry, the Imperial cavalry used the caracoll at the time, a rotating formation where the front cavalrymen would fire pistols to open up a gap in the pike formation - however, since the wheellock pistols used at the time had an effective range of about 10 meters and the pikes were about 6 meters, there was very little margin, and most shots were fired further away and thus very ineffective. Three ranks deep for pikemen was found enough to discourage the Imperial cavalry from charging head on, and if they resorted to the caracoll, the Swedish infantry had much superior firepower with both musketmen and supporting light artillery.
Sources:
Ulf Sundberg: Sveriges Krig 3, 1630-1814.
Göran Rydstad m. fl.: Historia king Trettioåriga Kriget (especially Michael Roberts part on Gustav II Adolfs military reforms and tactics).
Lars Ericsson Wolke, Göran Larsson, Nils Erik Villstran: Trettioåriga Kriget, Europa i Brand 1618-1648.
Lars Ericsson m. fl: Svenska slagfält.
Tom Gullberg: Krigen kring Östersjön, Lejonet vaknar 1611-1660.