I will go more in depth about your actual question, but first I'd like to write a bit about why your question is a bit fundamentally flawed

If I can paraphrase, your question boils down to "in the 1400s and 1500s the Ottomans were extremely powerful, and their military defeated most enemies that it confronted. By the 1600s wars were a more even contest, and at some point in the 1700s, the Ottoman military became a "laughing stock"

Your question essentially is predicated on the "stagnation and decline" narrative of Ottoman historiography. This theory posits that the Empire was powerful right up until its defeat at the First Siege of Vienna, and from there the Ottoman Empire entered into a long, drawn out period of stagnation and inevitable decline as the Empire fell behind Europe technologically. This was the conventional narrative of Ottoman history that dominated in the West, and remains influential in popular perceptions of the Ottoman Empire in the english speaking world today.

But it's a theory that has completely fallen out of favor among historians of the Ottoman Empire. The decline thesis only even superficially works if you look at an incredibly broad spectrum of history, and lump 300+ years of history into one narrative. When you examine specific beats within Ottoman history, like all history you are confronted by a much messier picture. For example, from roughly 1590-1610, a period usually still marked as being part of the Ottoman "golden age", the Ottoman military was fairly crippled by large scale revolts throughout Anatolia by non-Janissary soldiers known as sekban. In 1710-11, firmly in the "decline" (or in your words "laughing stock") period of this narrative, the Ottoman military decisively defeated the Russians during the Pruth River Campaign, and were in a position to completely destroy the Russian army and personally capture Peter the Great

But, on to your question. It is broadly correct to say that the 1600s saw the Ottomans no longer conducting sweeping, breathtaking conquests of Europe, the 1700s saw the Ottomans lose more wars than it won, and by roughly 1800, the Ottoman military was in a state of collapse. Why? It's difficult to generalize over such a long period of time (200+ years!), but we can identify a few key trends

1. European states caught up to the Ottomans - When the Ottoman Empire fought its incredible wars of conquest in the 1400s and 1500s, in many ways it was the most advanced military force in Europe (or perhaps the world!). The Janissary corps that other users in this thread have discussed more served as both a standing, professional army and a professional bureaucracy. While most European states were highly decentralized realms where Kings had to negotiate with squabbling dukes and lower aristocracy for support (both military and financial), the Ottoman Sultan could personally command the loyalty of a powerful and organized state. This simply gave the Ottoman military far more resources than anybody else, both in terms of soldiers and in money and resources to pay for those soldiers, pay for supplies, organize baggage trains, etc. But this status quo was not to last - European states began to centralize and professionalize, and soon the Ottomans' Hapsburg arch-rivals could command more and more financial resources with which to support more and more military resources. By the 1600s, Ottoman armies were now competing with Austrian armies on a relatively equal footing, whereas previously Ottoman forces had a significant material advantage over the Hungarians, Serbs, Hapsburgs, Byzantines, etc

2. European warfare became indecisive - Put simply, the 1600s and 1700s were a period of history where warfare became indecisive, wars of siege rather than decisive battle and sweeping conquest. European states could mobilize far larger armies than ever before, and defensive fortification technology advanced rapidly. Put together, these things meant that field battles were relatively rare, while sieges became frequent and drawn out. The Ottoman-Hapsburg wars were no exception - the Ottoman-Hapsburg borderlands were some of the most highly fortified places on earth, and wars between the two were affairs more of raid and counter-raid, siege and counter-siege, rather than decisive battles that wiped out the nobility of an entire country like Mohacs. This continued into the 1700s - prior to the conclusion of the Russo-Turkish War in 1774, territorial exchanges in both Ottoman defeats and victories were minor

3. The Empire's military and bureaucratic elite, the Janissary Corps, declined - /u/McGillis_is_a_Char goes into this more, but essentially, the backbone of the Ottoman state was the Janissary Corps, and by the late 1700s especially the Janissaries had become a corrupt, parasitic, and praetorian force in Ottoman politics. Muster rolls were enormously inflated to embezzle money, previously stringent recruiting and training standards had been almost eliminated, and the Janissaries used their enormous political influence to remove reformers and threats to their power from Ottoman political life (often fatally). At the same time that the Ottoman military and bureaucratic backbone was failing, European states were becoming ever more efficient at raising revenue and turning that revenue into effective military forces

There are other factors to consider - sheer logistics is one (it's much easier for the Austrians to fight a war in nearby Hungary than it was for the Ottomans to fight a war on its distant frontier, through hundreds of miles of roads running through rugged Balkan mountains), but these three factors I think sum up a very broad question

As for sources - one of the best scholars to read about the Ottoman military during the Empire's height is Gabor Agostan, who I had the pleasure to take a class with during undergrad. His Guns for the Sultan: Military Power and the Weapons Industry in the Ottoman Empire is overall about the Ottomans' military industry, but also contains a lot about just how the Ottomans organized their military and fought wars at this time. And he's written a number of articles about the Ottoman military and the overall Ottoman-Hapsburg conflict. In researching this question, I also learned that he actually has just released a new work that is a grand survey of the Ottoman conquests up to 1683, called The Last Muslim Conquest: The Ottoman Empire and Its Wars in Europe (I have not had a chance to read this yet, but if it's anything like his other works it will be an excellent resource)

For a general survey of Ottoman military history (and really, most Ottoman military questions), you can look at Mesyut Uyar and Edward Erickson's A Military History of the Ottomans: From Osman to Atatürk, of which there is apparently a free PDF avaliable (link not working because of reddit formatting - https://psi424.cankaya.edu.tr/uploads/files/Uyar%20and%20Erickson%20(eds_),%20A%20Military%20Hist%20of%20the%20Ottomans.PDF)

A caveat - Erickson has peddled in Armenian Genocide denial. That doesn't invalidate all the work he's done on Ottoman military history, but that context must be kept in mind, particularly when reading anything of his involving Armenians or the Ottoman war effort in WW1

A very popular Turkish historian nowadays, Emrah Safa Gurkan, discussed these topics in a youtube video. Some things to keep in mind:

1- Categorising the Ottoman history as ''rise, stagnation, falling'' based on conquests actually shows us how little we know about Ottoman history.

For example when did the first devaluation happen? (As early as Mehmet II period, 1445 summer).

First inflation: 1585.

So saying falling started after 1683 failed siege of Vienna actually means Ottoman history is not well studied so we look at the most superficial parameter of the success of conquests.

2- In 1744 population of France itself was equal to Ottoman Empires entire population:

https://www.reddit.com/r/MapPorn/comments/bbl8no/european_countries_by_population_in_1744/

Many people dont seem to understand this. Imagine France is smaller than even modern Turkey let alone Ottoman Empire of 1744. There has been three population centers in the Old World: India, China and Europe. So what we call ''great divergence'', the moment Europeans opened the gap and joined a completely another league from rest of the World, could have happened in China or India as well according to many historians but noone stands on the possibility of Ottomans making that. Because when you look at the accounts, the numbers, you dont see the urbanisation, fundamentals in the cities of Middle East and Anatolia. There are economically isolated cities, parasitic cities being fed by the Anatolian/Balkan villages.

So Ottomans were actually even in their golden age, even in their rising period; were actually fighting against the time.

3- You can think of Ottoman Empire as Soviet Russia in Cold War. Political and military power arent equal to good economy. Ottomans were always a political/military superpower never an economical power. Ottomans never had the numbers European kingdoms had. Europe made its revolutions both in civil life and in technology gradually as its urbanisation continued. Ottoman cities never reached to that urbanisation. There was simply no population, no fertility, no capital but that becomes overshadowed due to political/military power.

And when you dont have economy, urbanisation the other side has, even your political/military power roots from within.

4- Change in military paradigm in Europe.

You will realise that all the processes are actually linked to each other with this one.

As you may already know, in ancient world and the period starting with the renaissance(pike and shot period) infantry dominated the warfare.

In medieval ages however, it was the cavalry.

Warfare of a society actually represents everything. It tells about the economical-political structure of the society in question.

If you are nomads you are good at horse archery. If you are fishermen you are good at naval warfare etc. Raising soldiers from military schools is a concept that started in 19th century. Before that this was the way things happened.

Anyway, with the fall and collapse of Roman Empire, Europe actually lost a tremendous amount of urbanisation and economical power. In that period disciplined infantry formations of the ancient world left their places for armies with limited elite cavalry and an army of peasants following. Limited elite cavalry were the nobles who could afford horses and the peasants following were dominated by them. This was the feudal period.

But over the course of medieval era, it is actually Europe coming back to the point of Rome in economy and urbanisation sufficiency. You see the rise of disciplined infantry again. It means economy is again enough to suffice disciplined infantry armies.

Couple of hundreds of knights and shit ton of peasants following is a medieval army. It is an army of a poor society. An army of pike and shot period however, tens of thousands of disciplined infantry with standardised equipment and training, that is an army of a society with more economical development.

Battle of Keresztes you talked about. Nice. Do you know the historical context of that battle? It was the first time cavalry heavy army of Ottoman Empire for the first time faced a tercio-like formation. A pike and shot army. An army of disciplined infantry. Ottoman cavalry charged at the wall of pikes, expecting them to break like they did every time but this time they didnt. They werent a crowd of peasants.

So at the end, Ottomans had to recognise it was the age of infantry again and their cavalry heavy army was outdated. Janissaries, the household elite infantry troops, whose numbers were previously 6.000 and 13.000 at most in 16th century had to be inflatated. In a short period of time their numbers were increased to 30.000-40.000. Turks for the first time were recruited, the training quality deteriorated, discipline deteriorated. Because they were the musketmen, the disciplined infantry of Ottoman Empire. The closest thing to the western counterpart I talked about.

But like I said they were household troops meant to protect the sultan from any Turkish noble rivalry. Their job was to be the policework of Istanbul in peace times and to defend the sultan in war times. Ottoman Empire couldnt create its pike and shot armies like in Europe. It wasnt as rich. It wasnt sufficient economical and urbanisation wise. Ottomans by structure couldnt make ''civilian soldier''. So they tried to close that gap with janissaries but at the end they couldnt afford them neither.

So basically Ottoman Empire never reached the economical/urbanisational sufficiency of Europe. They had good ideas of modernisation and they did keep up with Europe as much as possible. But because they lacked the economical fundamentals behind it, their iterations of European warfare techniques could only be superficial. They still did fine until 19th century.

The thing is they were so bankrupt that they didnt have power to force down the constant revolutions in warfare from Europe to below. Though they managed to create a very elite Europeanised military staff cadre with their latest efforts and that cadre managed to carve out Turkey from an empire under occupation from all sides. They managed to establish a bureucracy. This bureucracy managed to make Kemalist reforms permanent. Iran for example failed in this regard. There wasnt a Europeanised bureucracy accumulation history like in Turkey. So the revolutions in there faded away.

"More specifically, I'm asking was there a real drop in the quality and organization of their armies or did they just fail to modernize enough?"

Neither, during the inconclusive Long Turkish War that you mentioned, the Ottoman Empire was plagued by Cellali Rebellions in Anatolia, which were not fully resolved until the early 17th century. They were also fighting a war against the Safavid Empire in the east. In both the war against Persia and Austria, the Ottomans found themselves operating at the peak of their logistical lines against empires that were more than capable of fielding armies of similar size and quality. As discussed in a recent book by Dr Georg B. Michels known as "The Habsburg Empire under siege" this static state of foreign affairs rapidly changed during the middle of the 17th century under the leadership of Grand Viziers Köprülü Mehmed Pasha and his son Köprülüzade Fazıl Ahmed Pasha. The balance of power in Hungary being decisively shifted in favour of the Ottoman Empire during the 1660s and 1670s. Georgetown University professor and Hungarian Ottomanist Gábor Ágoston noted that the Ottomans did not lag behind the Habsburg in weapons technology and firepower. Thanks to an efficient armament industry, "stockpiles of weapons and ammunition greatly outnumbered (and often doubled) the supplies of their Hungarian and Habsburg adversaries." Ágoston argued further that "Ottoman military superiority" was not broken until the 1680s, and then only by an international coalition that included the Habsburg empire as well as Venice, Russia, the Polish Lituanian Commonwealth and the German principalities in what became known as the "Great Turkish War." Historian Rhoades Murphey similarly noted the superior ability of the Ottoman state to mobilise men and resources for war. He concurred with Ágoston that "there was no hope of a successful military challenge against the Ottomans" before the pan-European military alliances of the 1680s and 1690s. As we can see, it would be foolish to attribute the collapse of Ottoman power in Hungary during the late 17th century to military inferiority, as opposed to fighting a multi-front war against several powerful European empires.

Furthermore, "decline" or "laughing stock" doesn't really work as a descriptor for the empire's army in the first half of the eighteenth century either. During the 1711 Pruth River campaign, the Ottoman army decisively outmanoeuvred its Russian counterpart and almost captured the Tsar himself, reversing the latter's conquest of Azov and thus winning the war. During the Ottoman–Venetian War (1714–1718), they successfully reconquered Morea and re-established themselves as the dominant power in the Aegean Sea. Between 1735 and 1739 they fought a two front war against both Austria and Russia, defeating the former and reconquering Serbia, Oltenia, Northern Bosnia and the southern Banat, ceding only Azov to Russia.

As we can see, the Ottoman Empire won just as many wars as they lost during the first part of the 18th century, and the economy was booming at least until 1760 or so. The 1768-74 war with Russia was the major turning point, which also coincided with economic contraction. Historians generally explain loss in that war as a result of three factors. First, the inexperience of the Ottoman army - the Ottomans hadn't fought a war in Europe since 1739 and thus the army was staffed entirely with new recruits. Second, the collapse of the supply system. One of the Ottomans' main organizational advantages over European armies in previous eras was their extremely sophisticated logistics, but again lack of recent experience in European warfare and the exceptional strain of the very large armies being fielded led to failures in the system. Third, a general economic downturn in the Mediterranean region starting in the 1760s and lasting into the early nineteenth century, which strained the Ottoman treasury during the war and increased the challenge of maintaining working relationships with the empire's provincial powerholders (the ayan households that, through cooperation with the central government, provided taxes and manpower during this period of "decentralization").

So the economic downturn and war with Russia initiated a crisis that would last into the early nineteenth century (1800s), when the reform efforts of Selim III and Mahmud II would succeed in re-establishing the centralized rule of the absolute monarchy while also greatly expanding the role and functions of the Ottoman state. This entailed higher fiscal demands on the empire in comparison with the relatively small and innocuous administrative apparatus of the 18th century, in which a great deal of power had been delegated to regional notables. To make a long story unjustifiably short, the enlarged 19th-century state was unable to muster the resources to match its new financial obligations, in large part a result of the need to remain militarily competitive in light of the ever-present Russian threat. The Crimean War of 1853-6 is often taken as the turning point for when the Ottomans lost the ability to finance the empire's wars without relying on European capital. So it's a highly complex issue. You have a much stronger state in the early to mid nineteenth century, but one that is less often fiscally solvent, as opposed to being technologically behind.

Therefore, I believe the late 19th and early 20th century defeats can be attributed to a number of factors, none of which are due to a lack of modernisation:

1. Diplomacy. Most European countries, particularly Britain, France and Russia had a vested interest in supporting independence movements across Ottoman territories, particularly in the Balkans and expanding their own influence there at the expense of the Ottoman Empire and other great powers in the region. An example of this was the Greek independence war of 1821. Similar to other European powers, when the Ottoman Empire was in a diplomatically advantageous position such as during the Crimean War, it was able to gain the support it needed to push back against the Russian advance. When it lacked this diplomatic support such as during the Greek-Turkish war of 1897, they were in fact forced to grant Crete autonomy through Great Power intervention despite winning the war militarily.

2. Fiscal insolvency and a small population. The Ottomans were successful in centralising the state bureaucracy and enacting various military reforms, but were often fiscally insolvent and found themselves fighting against opponents with a significantly larger tax base and population reserve than they had, an example of this being its later wars against the Russian Empire. During the Russo Turkish war of 1877-78, the Ottomans were equipped with Krupp manufactured breach loading artillery guns and Winchester manufactured lever action rifles. Adopting a defence in depth strategy for the defence of plevna. The result was a pyrrhic Russian victory. Despite taking the city, the latter suffered massive casualties and were soundly out-performed on a strategic level. However, the Ottomans simply lacked the reserves and finances needed to continue the war.

3. Logistics. Similar to earlier wars, the Ottomans were continuously fighting on multiple fronts. What had evolved into a moderately successful guerrilla war against the Italians in Libya in the Italo-Turkish war of 1912, had to be cut short because Constantinople needed to rapidly transfer her forces into the Balkans, in preparation for the oncoming first balkan war. The abandonment of the Libyan resistance forces resulted in Italy securing a victory and its first colonial possession. As discussed by Sean McMeekin in the Berlin-Baghdad express, 1914 was not the start of World War One for the Ottoman Empire. Their war for survival had already begun in 1911, and the Great War was merely a continuation of this. u/chamboz has discussed this elsewhere, but between 1914 and 1918, the constant need to logistically support large armies on multiple fronts meant that the Ottoman Empire once again found itself unable to sufficiently concentrate enough of its forces in a particular theatre and was thus defeated in detail, despite impressive military performance within particular theatres such as the Dardanelles and Kut.

Bibliography and further reading:

1. The Habsburg Empire under Siege: Ottoman Expansion and Hungarian Revolt in the Age of Grand Vizier Ahmed Köprülü (1661–76) by Georg B. Michels (March 2021)

2. The Last Muslim Conquest: The Ottoman Empire and Its Wars in Europe by Gábor Ágoston (June 2021)

3. Ottoman Warfare, 1500-1700 by Rhoads Murphey (1998)