Good question. I've written a bit about the comparison of Spanish versus English in view of the Armada here.
The Spanish came into that battle first of all wanting to avoid confrontation with the English fleet. Their objective was to meet up with Farnese's Spanish troops in the low countries: escort a large flotilla of transport vessels to the Low Countries where they could take the Army of Flanders across the strait to England in order to conquer England.
And this they nearly succeeded in doing, except for weather, English gunnery, and English persistence.
The two navies had very different goals. The Spanish would have much preferred to avoid the English navy altogether. But they had to stay together as a fleet to protect the slow and vulnerable transports. If they had to fight, they preferred to fight on the decks, not from a distance. The English wanted to stop the invasion. They didn't want to fight a boarding action, they preferred ranged gunnery action.
The Mediterranean had taught the Spanish how to fight as a navy, culminating in their victory in Lepanto. But that style of fighting failed in the battle against England due to asymmetric objectives, advances in gunnery, differences between Mediterranean and Channel conditions, poor high-level strategy (most agree that the Spanish will likely have done better if they had either invaded England or Ireland straight from Spain itself, or have the Army of Flanders do their own crossing, rather than have to escort transporters literally around England first), weather, tactics, etc.
At the Gravelines, due to attacks by English fire ships on the evening of Sunday August 7, the Spanish ships had to cut their anchor in order to sail away. By morning, most of the Spanish ships were still scattered all along the dangerous Flemish coast -- that coast had dangerous sand banks and reefs. At this point, Medina Sidonia still wanted to continue with the operation, he called on his commanders to re-group the ships.
When dawn broke on Monday August 8, the English saw the scattered Spanish ships and led an attack by 30 ships. To buy time for the rest of the fleet, the Spanish commander Don Alonso put together a small fleet of large galleons, to sacrifice themselves by delaying the english.
The next day, Medina Sidonia still had ideas to renew the effort, however morale was low, his strongest ships were either battered or lost, and most importantly were two factors: (1) in escaping the fire ships, the ships had to cut their best anchor, so it is now much harder to gain safe anchorage, and (2) a strong offshore breeze started to push the fleet off into the North Sea.
In that era, safety was gained by sailing together in tight formation. A scattered fleet is easy prey. Thus, the Spanish decided that the meeting with Parma's army was now impossible.
References
/u/Itsalrightwithme has covered the basics here really well. I'd just add one more thing: the Spanish ships were running out of ammunition for their light guns and small arms, and while they still had plenty for their heavy guns, the English persistence in gunnery didn't allow them to reload.
The English were averaging about 1 to 1.5 shots per gun per hour; the Spanish averaged about 1 per gun per day. Both navies had to reload their guns over the side of the ship at this point, and the English could take advantage of their ships being more nimble to draw off and reload, while the Spanish ships were more or less constantly being battered. Given that the basic plan for the Spanish ships was to board and overwhelm the enemy with musketry, that represented a significant loss of their fighting power. They were also now downwind of the invasion army, and would have to fight their way back upwind to get back to Parma. And, they didn't know the English were low on ammunition; the English ships followed them into the North Sea "setting on a brag countenance" to hide their weaknesses.
As to why Medina Sidonia didn't just make port in Scotland or somewhere else friendly (Hamburg, Denmark or Norway), replenish and try again, we really still don't know. We do know that Medina Sidona's heart really wasn't in the battle to begin with, and it had been delayed time and time again.