To start and simply put, paraphrasing Felipe II it had been a matter of "fighting the elements and the men". Expanding that thought:
The Spanish Armada (ironically known as the "Armada Invencible") was composed of larger, war-like ships that were suited for carrying a large number of soldiers rather than quick maneuvaribility at sea.
This disadvantage combined with several adversities, such as the Armada breaking formation and several ships losing their anchors early on to English attacks (with flaming boats known as brulotes) prior to reaching the North Sea and unexpected weather conditions deviating ships from their original route from Dunkerque (currently France) where they had docked to pick up more troops towards the Netherlands rather than British soil. By then, the vessels were damaged, the possibility of a swift attack lost with virtually no viable organization between ships, so when the wind abruptly changed a few hours later several commanders took the opportunity to head towards open sea in orden to at least reduce the advantages of the English side.
Note that by this point the focus had already definitely moved from "invasion" towards "retreat and live another day". Why, then, did they go around Britain rather than retread their route? Esentially there are three main reasons:
Open sea sailing, even following a coastline, was a well-known process that the sailors had experience with, and could better predict further obstacles and follow expected winds.
As discussed, even fully furbished the ships involved had been too unwieldy. By this point they were nowhere near as encumbered, but they had suffered damage to anchors, sails and other elements that further reduced their movements, so a wide arc was much more within the realm of possibility than a "tight" turn.
Last but not least, the English were still in pursuit. While some commanders had favored the suicide run, and others called for surrender, eventually the decision was made to return to Spain and the safest route to do that was to avoid the heavily stocked docks near la Mancha (closer to France) and go around Ireland and Scotland, following then the currents until reaching a Coruña.
There were further battles and further problems, including another tempest, but the bottom line is that the decision to go around Britain was, while dangerous, still possessing the greater chance for survival.
SOURCES (All in Spanish, apologies): https://www.google.com/amp/s/www.nationalgeographic.com.es/historia/grandes-reportajes/la-armada-invencible_6643/amp
https://www.menudaeslahistoria.com/los-barcos-de-la-armada-invencible-con-nombres-y-apellidos/
https://www.google.com/amp/s/www.armadainvencible.org/la-supuesta-derrota/amp/