Considering during the Rennaissance Western Europe began to gain more and more access to classical works, and this was the time that pike armies began to dominate the battlefield, did any commander ever use ancient works on Alexander the Great and his Macedonian phalanx to try and gain some insight on how to use said armies effectively?
Not exactly, because it wouldn't have made any sense to do so. Firstly, the difference between theory and practical applicability is large. A lot of ancient treatises existed and were broadly known, like Vegetius' De Re Militari and Frontinus' Strategemata, but they lacked the practical aspect of individual soldier drill and unit manoeuvre. Military men could see that well trained and drilled soldiers were useful. The large pike armies weren't a rediscovery of ancient ways either, it was an evolutionary process and the Spanish had developed their own body of modern military knowledge by 1580-90, mindful of the classics, but based on modern ideas. There is a clear evolution from the medieaval spearman/foot man at arms in armour and armed with poleweapon through Swiss halberdiers to Swiss pikemen so admired and copied by other forces.
Several at the time authors did write on the subject, e.g. Machiavelli in 1521 Libra della Arte della Guerra and Justus Lipsius later in 1595 De Miltia Romana libri quinque, Commentarius ad Polybium, seeing the similarity of the large Spanish tercios to Alexander's Macedonians and proposed that copying the Roman style that defeated the phalanxes would be a good idea. What the military reformers needed was Claudius Aelianus Tactica and Emperor Leo VI's version of Tactica. In reality we should think of a correlation not a causation between the translation and distribution of ancient military texts and the development of the pike and shot armies. The Renaissance saw the similarities, and considered the ancient sources, but were not bound by them as they could easily see their realities were not the same.
Secondly, the firepower a modern army of the late 1500s possessed was orders of magnitude more powerful that what the ancients had which limited the usefulness of ancient manuals and even some of the earlier contemporary material. What we today then refer to as pike and shot tactics are basically the reforms pioneered by the Dutch and Maurice of Nassau (and then further refined by others) to create a more uniform army trained to the same standard, using the same equipment and smaller composite units with more tactical flexibility compared to the Spanish system.
Keith Roberts, Pike and Shot Tactics 1590-1660 (2010) goes into more detail on the subject.
So your question hits on an interesting topic in this field which I'm going to sort of expand on a bit. As with many historical questions, the answer is a bit of yes and no, so bear with me. Now to start off, my research in this field primarily deals with the Dutch experiences in the Eighty Years' War so much of what I'm talking about will come from that.
Now on the topic of ancient texts and study of military in antiquiety and the role it played in early modern warfare, one thing we have to be clear about before anything else is that the majority of military leaders in this period were massive classics nerds. Maurits van Oranje studied this heavily at Leiden for example, and Roman thinkers such as Vegetius and his accompanying De re militari were favorite texts among generals in the Empire, in France, and, of course, in the Dutch Republic. Much of the drill surrounding the States Army in this period was, on paper, and as attested by its leaders, inspired by the Roman style, and others in the "Greek style" which in this context tended to refer to phalanxes and organizing pikes in a likewise manner.
To this, a lot of the development that went into just how the States Army looked and functioned as such was a matter of public debate. There were public drills and tests between conflicting officers frequently, our man Maurits took part in dozens of these, and the debate often surrounded which classical insspiration was a superior one, and if that was a valid interpretation at all. Now in considering this, we also have to deal with it critically and also recognize that styling something after the Romans or claiming origin in Rome was a very common tactic in Europe at this point to lend legitimacy to it. When we look at the drill of the prince of Orange and how he organized his men, we do not see exactly a replication of the Roman style of fighting. Indeed, there were thinkers at this time who advocated it, there was even advocacy in one circle to organize an army based around a maniple system that broadly ignored firearms, and yet as the armies took hold, we really don't see an exact replication of Rome, we see an organically thought up force that bore perhaps a cosmetic resemblance to Rome and this was used to further legitimize it.
Lipsius, for example, argued that the Phalanx was always defeated by the Romans when in formation and in tight drill and advocated that commanders ought emphasize pikes over firearms, another classicist at the time, Scaliger, argued that firearms rendered Roman thinking void as a whole. There was a lot of debate around this time about it, almost as much internal argumentation as there was actual fighting with the Spanish, yet despite this we often see more an argument as to a recreation of Rome with a (at the time) modern ideal than actual Rome. One of the more notably curious claims made by Maurits and his cousin was that the Roman drill as reflected in Vegetius originated the concept of the countermarch. It's a bit dubious, but to bring it back, it was entirely possible that relatively new concepts in this period were claimed to be Roman in style so as to lend credibility to it.
When we actually look at the typical Dutch unit from this period, we make a few notable examinations. The lowest infantry combat unit in the States Army was the company, regiments existed as administrative units broadly in that army so the majority of stuff was at the company level. The three principle components of a company were spiessen, pikes, musquetten, muskets, and roers, calivers, a kind of standardized arquebus, lighter than a musket and fired freehand. Few drills for the calivers and muskets would resemble anything of a Roman army, the focus for them was fighting as part of the company and in a greater order than one might find in skirmishing drills in Vegetius or Polybius. Looking at pikes, similarly, there isn't a lot of similarity beyond some aesthetic commonalities. These drills, mind you, would be taught by captains to their companies after the drills instructed to them for training their men by their Generals, in this case the prince of Nassau. The actual praxis of the insights gained from Roman and Greek thinkers did not ultimately play much a role in how a fighting unit was trained and took shape.
Now, this doesn't mean they didn't look for insights on that. Again, especially in the 1590s, there was a lot of public and private discussion of the classics as far as how it might affect the armies in that time. Willem Lodewijk and his cousin Maurits van Oranje very famously discussed the Battle of Cannae in depth while the two were having public displays of experimentation regarding their new infantry formations and drills, they even ordered new translations of accounts of the battle purely for this reason. On top of Vegetius and drilling, there was a vested interest from thinkers at this time in just how the people they looked to had actually performed in the field and how they might replicate orders of battle and specific methods of fighting. However, back to Lipsius's claims, in the end, very famously, the States Army as it found itself at the end of the 16th century was one that saw a drastic uptick in the amount of firepower per company, this was something Maurits and Willem specifically put in place around 1598 and it played a very important role in the campaigns that would immediately follow it.
So, to sort of synthesize all of this, yes, many military leaders in this period looked to Rome and to Greece for inspiration and insight into how they might better fight a war. Much of this dominated public and intellectual debate as to how an army might be raised, organized, and led. When it came to it, little of directly Roman or Greek thinking actually went into how an army was organized, however, but there were some trends across history that were still present. The focus on distinct and standardized training and the presence of many well trained men forming the core of an army rather than emphasizing a few key units was one aspect of military thinking from this time that many claimed as inspired by the Roman model. Furthermore that very drill and marching was said by many to be inspired by Roman texts. I'd be a bit more critical of this and repeat the point that I think a lot of this was more an attempt at providing credibility and legitimacy to one's ventures in an era where much of the intellectual milieu was focused on looking to the wisdom of Rome and the many Greek states. More than any practical inspiration, a philosophical one, to be sure.
This answer has primarily drawn from the following:
Limits to Revolution in Military Affairs by Geoffrey Parker, a very good paper on the organizational switch up of the States Army in the leadup to the Battle of Nieuwpoort. Aspects of Parker's "military revolution" thesis have been criticized and he discusses this a lot here, notably, its limits, as the title says, and aspects of it that were eurocentric in previous editions as well as taking an operational deep dive into the performance of the Dutch army in its semi-reformed state. I've also taken from his dedicated book on the Military Revolution Thesis but not quite as much, it's aged a bit but his analysis of the formation and training of a Dutch infantry company as opposed to a Spanish regiment (tercio) remains useful.
Furies by Lauro Martines, while a general work in some capacities, an overall very good look in my opinion at the ways in which the military interacted with the state and the people and vice versa, having many chapters devoted purely to the interactions between soldiers and civilians in a period where that was intensely blurred. He also devotes a lot of time to discussing organizational and logistical shifts across the period.
Wapenhandelinghe, van Roers, Musquetten, ende Spiessen, Weapon Handling of Calivers, Muskets, and Pikes, loose translation. This is a drill manual with some engravings, published in 1608. Scans of it are available online, but it's in ye olde Dutch which is a bit hard to read if you aren't familiar with it. Translations exist and will do a decent enough job but all of the ones I've seen make a lot of bad translation choices that I don't agree with, I might just make my own translation one day to combat this, but anyway, this provides information as to drills of Maurits van Oranje as of 1608, much of these having been set in stone around 1598 and codified for the entire army around 1600.