This question is in two parts - I'll answer the second part first as I hope it will make better sense this way. I imagine I'll brush over a lot of history and terminology so please do ask if anything is not clear. I also apologise if I drop in random names and French phrases - I'll try to clear these up once posted to make this more readable! My sources will be listed at the end of this post.

"How did they fail so fantastically to get ahead of their finances?"

This is best answered by looking at why the government failed to reform more generally, which can largely be attributed to the actions of the monarchy itself. I want to take a long-term approach which doesn't just look at Louis XVI, but also his predecessors, in order to explain fully.

1. The French economy in the long term: France, unlike England, failed to establish a central bank. This was in part due to the failure of the Law reforms of 1719-20, when attempts at a semi-national bank ended up precipitating a financial crisis. I'll provide more detail on the impact of this in part 2, but basically this meant France did not have a cheap way in which to channel public debt. Hence in the 1780s, a substantial amount of revenue was put towards addressing debt (43% according to the infographic in the source you provide). This is representative of the French economy in the eighteenth century, which in many ways failed to adapt and modernise to changing conditions. France in the eighteenth century saw two largely separate economic spheres: Prosperous coastal regions and their limited surroundings, and the traditional, unreformed interior. The manufacturing and agricultural sectors in the latter failed to progress in the eighteenth century, so much so that Blanning has claimed that “the French economy of 1789 was essentially the same as the French economy of 1715”. In the long-term, the French economy was not especially strong by the time Louis XVI came into power, which undoubtedly impacted on the government's financial situation.

The economy was also badly damaged by the disruption of colonial trade. This was because of...

2. Warfare: This was a particularly prominent frustration because of the regular and massively costly wars under Louis XV and Louis XVI. Riley has argued that the height of the parlement’s power coincides with times of great financial difficulty due to warfare; the government, in order to gain acceptance for high taxation, was forced to cooperate and give concessions to the parlements. As Turgot, minister of finance, said to Vergennes, French intervention in America would make impossible “changes crucial to the prosperity of the state and the relief of its inhabitants”. By 1763 financial matters were already serious, but it was French intervention in the American War of Independence that led to a truly desperate financial situation. Yet at the same time there was a complete failure to reform the Crown’s finances. The government was unwilling to shirk from its international commitments, the costs of which greatly outweighed economies made in minor areas such as the royal household. Taxes were raised, such as the introduction, doubling and later tripling of the vingtième. But the parlements strongly resisted further increases, so much so that Necker resorted to borrowing to fund the American War instead of attempting to raise taxation further. Financial planning was also very difficult because the government never knew how long each war would last, and so a financial strategy was near-impossible to accurately create (along with the unpredictability of harvests, trade, etc).

This was in the context of relative peace following the death of Louis XIV (1715). War began in earnest in 1733 over succession to the Polish throne. The War of Austrian Succession (1740-8) and Seven Years' War (1756-63) were both massively costly major international conflicts. This was followed by the American War of Independence (1778-83). This also brushes over many smaller conflicts, and periods in which France was forced to mobilise and ready itself for war. Louis XVI even started preparations for the American war shortly after the Boston Tea Party. To quote Félix: "If one must seek an overall explanation for the financial crisis of the ancien régime, only war - and the policy of France's grandeur - can provide it". I will expand on the costs of this in the following post, but for now, it should suffice to say that warfare meant the monarchy was incapable of reigning in its finances.

3. The parlements: The greatest institutional resistance to reform undoubtedly came from the parlements of France. Louis XIV (14th) was able to avoid this opposition to a large extent, but upon his death the new regent Phillip d’Orléans revoked the law of 1673, allowing parlements to remonstrate before registering laws, a legal version of grievance before supply. The parlements were therefore restored to their traditional role of opposition to the government, a judiciary with a political, even constitutional, role. The lack of representative institutions in France meant that the parlements saw themselves as giving consent to governmental policy. This gave them limited authority on all aspects of legislation, from religion to finance. Although in the face of remonstrances the king could issue a lit de justice to force legislation through, this was rare as controversial legislation was harder to enforce in practice. By 1750 the Parlement of Paris was even questioning the authority of the lit de justice. On the question of religion, the parlements strongly rejected Louis XV’s implementation of the Unigenitus bill, and under Louis XVI exerted so much pressure that he was reluctantly forced to expel the Jesuits from France, demonstrating the influence the parlements had on government. The parlements were therefore a clear obstacle to any reform in France, willing to oppose much of the monarchy’s efforts. Financial reform was impeded by the parlements.

Yet the influence of the parlements must not be overstated. The parlements were not united, between one another or even internally. Deep divisions between bodies meant their opposition to the government was far weaker than it potentially could have been, with opposition often short-lived and/or uncoordinated. Relaxations of the grain trade (1763-4) were strongly condemned by some parlements, but hailed by others. Under Maupeou as Chancellor, the parlements were even dissolved. The government faced little trouble in doing so, and some parlements willingly cooperated in Maupeou's new courts. Further, and more importantly, Doyle has rightly argued that the strength of the parlements was inversely related to the strength of the government. Louis XIV encountered little resistance after 1675 because the parlements knew any serious opposition risked a harsh response from the government, such as the exiles of the parlements of Rennes and Bordeaux. Under Cardinal de Fluery (d. 1743), and Maurepas (d. 1781), times of united government, the parlements mounted little opposition. Yet during times of weaker and divided government, such as between 1750 and 1771, opposition from the parlements was very strong indeed, with the Parlement of Paris even sending “princes of the blood” invitations to their sessions in order to frighten the government.

Hence, a weak government led to greater resistance from the parlements. Which leads to...

4. The weakness of the monarchy itself:

4a) Factionalism: The resistance of the parlements was massively exacerbated by the monarchy's sponsorship of factionalism in government. Divisions between ministers fomented opposition in the parlements. Whereas their predecessor Louis XIV sought to limit factions, Louis XV in particular strongly encouraged it following the death of Fluery (1743). Unlike the ministers of Louis XIV, who were willing to prevent the judges wielding political power, Louis XV found that ministerial interference in the parlements likewise meant the parlements increasingly interfered in ministerial and court politics. Louis XV was motivated to do this because he wished to avoid being influenced too heavily by any one single party. But the consequence was complete inconsistency in government, and limitations on financial reform put in place by a confrontational magistrature. This is highlighted by the example of Bertin, who incurred the disdain of the parlements (encouraged by his rival Choiseul) after refusing to revoke the unpopular wartime levies. This disdain let to his fall, as Louis XV was unable and unwilling to sustain a minister unable to maintain the support of the parlements. Under divided governments it was easier for parlement to resist changes, as rival ministers used the parlements as forums for opposition to the government: a minister who opposed a particular policy would take the side of parlements attacking it, knowing well that the monarchy was disinclined to conflict with the judiciary. The rivalry between Machault and d’Argenson is a typical example of how ministers used the parlements to stir up opposition to government policy, the experience of Bertin against Choiseul another. So, Louis XV and Louis XVI incited divisions in their governments, which meant that the parlements gained greater authority to resist changes.

I'm close to going over the word limit, please see the replies for the continued answer - my sources will be cited at the end.

There are a few attenuating factors as to what made the situation in France untenable at the time. One of the main points being that money is no substitute for food (especially in an early modern world economy just barely globalizing) so any theoretical wealth in France could not be readily liquidated into goods and services.

However, I think if you want to start looking at sources I would recommend the primary source of the Compte Rendeu (which should be, at least partly, translated). This will explain the crowns mistaken financing of their own debt if you see how much cooking the books went into that report which effectively ignored the heavy outstanding 'extraordinary' loans to paint a rosy picture of the annual revenue and budget that was swamped with paying the interest on the borrowed principle.

There was also the policy of 'useful splendor' in which 'frivolous' spending was endorsed and increased by the finance minister (the one between Necker's terms) in order to reduce the interest rate on their mass of existing and future borrowing needs thus paying down principle and interest with lower interest loans to slowly rebuild the health of the balance sheet. This policy backfired due to ministerial shuffling in the position of the finance minister (interesting link that the question of the king and his ministers will also bring down Charles in 1830) and due to not being very sound as a concept in general.

Lastly, I would emphasize the government of France was legally limited in some ways that hadn't been a problem since his grandfather's (Louis XIV) time. You can compare this as to maybe the idea that we might need a constitutional amendment to fix a particular political problem though this has been done very infrequently in the last half century. This meant that the King wanted to avoid any changes of government manner at all costs (particularly the costs of high debt) as this would have involved consulting bodies that necessarily were primarily composed of his opponents in large part, namely the regional parliaments and the Estates General.

Sources are tricky because the historiography of the French revolution itself has been a force in history via different attempts to explain what happened. I like Tackett's writings insofar as they cover the pre-revolutionary period you seem to be interested in. Francois Furet might also be a writer of interest for you but it's hard to recommend a trusted survey historian of depth.

/u/frederfred1 covered much of what I wanted to say much more comprehensively, but I started this a couple of hours ago and figured I'd finish the post -- what the hell. The main part of understanding the problem is that it was a vicious cycle of increasing debt that necessitated an unpalatable (to the King) solution which lead to greater failure. A sort of catch-22 for the monarchy that would have only hurt its power any way it chose.

I am on mobile, so apologies that some dates and numbers are generalities, but it feels like nobody has discussed a huge component in the Haitian Revolution, which is partially responsible/overlaps for the conditions of revolutionary France. At its height, the island was producing 2/3rds of all of France's exports, and when the country was lost a major revenue stream for France was disrupted.

In fact, Haiti is also partially responsible for the defeat of Napoleon, as Napoleon sent some 300,000 troops to the island to recapture it and failed ... which was happening at the same time ash is failure in Russia.

Fun fact: the major reason the Louisiana purchase was sold to the U.S. was because that land was going to be used to supply San Domingue (Haiti), but once they lost the island Napoleon sold it off (also because he were cash strapped).

Source: Avengers of the New World by Laurent DuBois