This answer will focus primarily on the British and Empire mounted forces. This is a really wonderful question!

Part 1: Officers and Leadership

Cavalrymen would not agree with many modern assessments that have been made of the arm. In fact, by the time the First World War started, British Cavalry were thoroughly modernized and proved able to adapt to changing battlefield conditions. This was important as, during the First World War, there were no other arms which had the mobility and speed of cavalry which enabled them to work effectively in reconnaissance + screening, rear and vanguard action, flank security, and a number of other related jobs. Until the advent of faster and more durable mechanical vehicles, there was no reason for Cavalrymen to see themselves as outmoded. Looking both forwards and backwards in 1937 Leon Kromer, then chief of U.S. Cavalry, wrote “In the World War, the most recent war of magnitude, Cavalry, in all theaters of operations, proved essential to a balanced combat team.” Of course, Kromer was not just referring to horsed cavalry, but also the emerging mechanized forces.

Prior to the First World War there had been hot debates in the UK over the role and future of the Cavalry. Generally speaking, critics of the cavalry wished to move towards a “mounted infantry” force - that is infantrymen who effectively just use horses to get from Point A to Point B and away from “cavalry” proper. There were some that struck a bit of an “inbetween” position, which advocated for “Mounted Rifles”, which were essentially cavalry and had all of the duties associated with that role - just sans a sword or lance. These debates led to the abolishment, then reinstatement of the lance, and a whole lot of venom being spilled. In the end, the advocates of a “Hybrid” cavalry won out.

So what was a Hybrid cavalry? This was a vision of mounted forces which could both dismounted rifle fire (and eventually machine guns) as well as mounted melee action. It was a vision of a highly trained and professional group of cavalrymen who, while being more lightly equippe, who had effectively mastered both infantry and mounted combat. One of the “hybrid” doctrine’s earliest proponents was none other than Douglas Haig - often marked as a ‘backwards’ cavalryman who thought of nothing but the charge. However, the doctrinal writings that helped shape in those Cavalry debates, in the form of Cavalry Training (published in 1912) paint a very different picture of his beliefs:

It is, however, by no means necessary when an attack is made that only one of the two methods [fire action with rifles or shock action with melee weapons] should be employed, for fire action can create favourable opportunities for sock action, and a well-executed combination of the two methods will often present the greatest chances of success.

Cavalry must be prepared, therefore, to use either the sword or the rifle, or the two in combination.

Here we see an expression in the training manual, and thus doctrine, that the British Cavalry went to war with in 1914 was one that understood that cavalry had to be an integrated fighting force, which could not rely solely on shock action. That is not to say that shock action never occurred, far from it, it was generally more frequent than most “pop” history of the First World War would have you believe, and was also generally successful. Cavalry Training, beyond being a simply a rote listing of drill, maneuver, and doctrine, makes a case for the continued existence of the arm:

An efficient cavalry is of incalculable value to an army [...] Without cavalry the other arms are hampered by ignorance of the eney’s movements, cannot move in security, are unable to reap effectually the fruits of victory, and have great difficulty in extricating themselves in case of failure.”

At the highest echelons, cavalrymen viewed themselves and their arm as an important part of the army. Their role during the war didn’t diminish this, and in some cases it even helped make their case, the U.S. Cavalry school’s History of Cavalry During the World War written between 1922 and 1923 proudly proclaims that “It has been proved that cavalry, whether used for shock effect under suitable conditions, or as mobile infantry, has still an indispensable part to play in modern war.” This is not just a post-war justification for the continued existence for the arm, it was the feeling of senior cavalry leadership during the war. Look again at Haig, who is often pegged as the arch-cavalryman, eternally stuck in the past, and unwilling to learn (while ignoring his efforts and enthusiasm for tanks, as an example). Haig talks about Cavalry in his diaries, although far less frequently than you’d be led to believe by many historians of the First World War who take almost every opportunity they can to remind readers that Haig was a cavalryman, and that he was dumb and backwards for seeing any role for the cavalry.

This is how Haig talks about cavalry in his diaries when he does. On March 14th, 1917 he wrote the following (emphasis is in the originals):

If successful, at Arras, exploit with all reserves and the cavalry.

If not successful, prepare to launch attacks near Ypres to clear the Belgian coast. All cavalry will be required probably if this attack is successful”

Haig brings up cavalry when he’s discussing the planning for Third Ypres, and beyond that his mention is that they will be used in exploitation (so after a breakthrough, rather than being used to create one) only after the success of Infantry at either Arras or Ypres.

During the Battle of Cambrai, later that same year, Haig simply reported that he had heard a report from Major General Mullens on the performance of the 1st Cavalry Division, and that Mullens was happy with their performance and that the battle had been “worth it” for them. These are the sorts of contexts in which men like Haig most often brought up cavalry, in essentially noting that Cavalry forces were to be used, or in reports about their usage. There’s not a lot of pontification, mainly because it is his diary, but it is interesting to note none the less. He’s far less “obsessed” with cavalry than his critics would give him credit for.

Much of the material I’ve quoted here comes from materials that were ultimately arguing for the continued existence of the cavalry arm (both US and British cases). They speak to your question, however, because it shows that they did not believe their arm to be out of date and useless (an assessment I ultimately agree with). This is what those in charge of cavalry believed about their arm - that it could function on the modern battlefield, and they utilized examples from recent conflicts to support their case. They were not against modernization, but a century on it’s quite easy to read technologies backwards and to see those who did not adopt first as “stupid” because they didn’t see what the future brought. It’s important, however, to remember that technologies like Airplanes and motorized vehicles were new, untested, and not fully reliable, and were not created in a vacuum. They were being made to sell for a profit, and those who were looking to profit off of their machines often did dress up the real capabilities of their technologies. Being skeptical of salesmen is a good thing, and someone is not stupid because they were cautious and happy to adapt with technology, rather than buy the sales pitch outright.

If we look at how British cavalry reacted to the development of the airplane, it wasn’t even close to an antagonistic relationship. British cavalrymen, on the whole, were more than happy to cede even some (if not all) of their reconnaissance duties to the airplane, and hoped for close cooperation with airplanes integrated into cavalry divisions (a concept which sounds very modern!) That is not to say this was a universally held position, but it was one that was echoed enough to warrant comment. British cavalrymen, on the whole, were okay with new technologies and felt that cooperation with them made for a stronger cavalry force (by a recofusing of what the cavalry’s mission was).

During the war itself, it is not as if airplanes supplanted reconnaissance entirely. Even during battles where we don’t think cavalry traditionally played any sort of role, they were there! For instance, during the capture of Vimy Ridge, Canadian Cavalry played a reconnaissance role which was well appreciated by contemporaries. Corps Commander P.B. Radcliffe & Major General W.D. Anderson both noted that patrols of the Canadian Light Horse’s reconnaissance was extremely valuable that day, and according to Stephanie E. Potter: “cavalry reconnaissance was far more accurate than aerial, infantry, or tank reconnaissance”. Within the BEF’s structure, there was a feeling that cavalry was still very valuable, and that in practice these beliefs were borne out.

Cavalry in basically every country, to include the highly motorised US, didn't consider the horse an 'obsolete' technology by any stretch of the imagination in the interwar period, WW1 notwithstanding.

First things first. Cavalrymen weren't stupid. They knew full well that charging machineguns was a really bad way of doing things. In many countries, that sort of 'charge into a defended area' was never a primary function of cavalry anyway. So the image of horses charging forward to be mown down by machineguns as the future of the branch was about as viable as the image of infantry charging forward (more slowly) to be mown down by machineguns. Infantry didn't go away because machineguns could kill them, and neither did horse cavalry.

What cavalry were supposed to be good at was reconnaissance and exploitation, and in the early inter-war period, there still wasn't anything capable of doing that job as well as a hippomobile unit. Tanks were, quite simply, too slow and unreliable, and armored cars too limited by wheeled mobility. There were all sorts of places, from the US's Mexican border to the forests of central Europe, where horses were deemed better suited for the purposes of such operations. Horses would be quite commonly found in many nations up until 1939 as a combat arm, the US was one of the final hold-outs.

It's also worth noting that not all of WW1 occurred in trenches on the Western Front. The experience of the Desert Mounted Force under Allenby, for example, was very productive and successful. For example, note the result of the Superior Board created to assess the lessons of WW1, which concluded that the American concepts of what cavalry was and how it should be used, both operationally and tactically, were sound (which, for the record, didn't involve charging with swords drawn very much). It also observed that there were occasional moments where they regretted not having a cavalry unit or two in reserve in order to exploit a breakthrough. Tanks were generally an infantry support weapon, not an exploitation weapon. Whippets like Musical Box notwithstanding.

It's not that the cavalrymen were ignorant of the possibilities of new technology. They were very well aware that the motor and the wing could greatly enable them to carry out their role, but most did not think that they could replace a good unit of horsemen until probably somewhere around the mid 1930s. Instead, these new systems would be used in conjunction with the horse (and in some experiments, used to transport the horse) to provide a more capable overall unit.

Things really started to change in the 1930s with the development of armored vehicles which were reasonably fast and reasonably reliable. They began to prove more capable in most circumstances than horses at doing the cavalry job, and thus the move began to replace horse cavalry units with mechanised ones. It's worth noting that the Russians still had pretty reasonable success with their horse units in WW2, though.

The thing is, horse cavalry was not obsolete in 1914, nor in 1919. It played a very large role in WWI, and in WWII as well. In fact, Portuguese horse cavalry was tactically crucial in the Angolan War of Independence, as was Rhodesian cavalry in the Zimbabwean Chimurenga, both in the the 1970s.

In the second half of the 19th century there was if anything a great revival of the capability and importance of cavalry. Organization generally transitioned from specialized types designed for specific missions (shock action, reconnaissance, mounted infantry, exploitation, security and liaison) to a more generalist type trained and equipped for all these missions. It turned out that some of the innovations that had limited cavalry’s effectiveness in the shock role in the first half of the century, also increased its effectiveness in others. The long range firepower of fast loading rifles, which made cavalry charges against prepared positions difficult at best, also increased the dismounted effectiveness of the mounted soldier. John Buford’s 1st Cavalry Division demonstrated this on the first day of the Battle of Gettysburg, by identifying, taking and holding the high ground south of the town, in the face of vastly superior Confederate infantry and artillery. The speed and firepower of the newly reorganized US cavalry held the Confederates at bay until the bulk of the US forces could catch up and deploy appropriately. This combination of mobility and firepower proved even more effective on the open plains west of the Mississippi, and with the advent of smokeless powder, crucial for both sides in the Boer Wars. So going into WWI, both sides, the Germans with their Franco Prussian war experience. and especially the British with their South African experience, had reason to believe horse soldiers would again be a decisive arm. That said, infantrymen had been predicting the demise of Cavalry at least since the destruction of the Russian cavalry at Balklava in 1854. Visionaries in 1815 pointed to Waterloo as proof that Cavalry was helpless against entrenched infantry. But experience in the US and Africa had shown that while the cavalry charge was now a niche tactic, the arm overall was still invaluable. I would argue that European cavalry was at its peak of efficiency in 1914, at least until the advent of Soviet Cavalry Mechanized groups in the middle of WWII.

Indeed the first three months of WWI in Europe saw extensive use of cavalry in traditional cavalry roles. The first shots fired on the Western front were between probing and screening German and French cavalry units. After the race to the sea, when the Western Front had largely settled into static warfare, the Germans executed a number of large cavalry raids into Courland, to the north-east. On the Eastern Front, where the distances were too great for even he enormous armies of that war to implement unflankable linear defenses, Cavalry remained the decisive mobile arm it had been before the advent of accurate, fast loading infantry weapons. At Schadow German cavalry flanked and broke the Russians, enabling a major victory in the Carpathians. Likewise Russian cavalry performed well in the Brusilov offensive in 1916. Back in the west, in May of 1918, The French II Cavalry Corps covered 135 miles in three days to cut off a potential German breakthrough in the Ourcq valley. Then when the Allies themselves broke through at Amien later that year, it was cavalry that reprised its traditional role as an exploitation force, chasing down and destroying the Germans and their reserves as they withdrew. Tanks of the day had nowhere near the speed, range, reliability or ergonomics to accomplish this, and they were vulnerable if not accompanied by infantry. While the war did not see much shock action, with saber and lance armed horsemen crashing into massed infantry, the ability to cavalry to cover long distances over difficult terrain and bring potentially decisive firepower to bear at critical junctures, was still unmatched. Motor transport existed, even tanks after 1916. But it was nowhere near efficient, available or reliable enough to replace the horse in any of its roles. That was even more true in the Middle Eastern theater, where open terrain and smaller armies dictated mobile warfare. They type of warfare at which horses and camels excel.

The two cavalrymen whom your armchair historians probably denigrate most, Sir John French and Field Marshall Douglas Haig, were actually key reformers who helped recreate the British cavalry as the finely tuned weapon of war that initiated the Great War in reconnaissance, and ended it in pursuit and exploitation. Haig was not only a major proponent of arming cavalry with machine guns, but a driving force in Britain’s prescient adoption of the light machine gun. He essentially wrote the 1904 training manual that shaped WWI British cavalry tactics. French was prominent in the faction behind the 1909 regulations that again formalized British cavalry’s role as highly mobile mounted infantry, primarily armed with long range rifles, but still capable of scouting screening and foraging, and also retaining a shock capability for exploitation and pursuit. Honestly, I could never approach the quality of this account of British cavalry on the Western Front by /u/jonewer/. Especially not the detailed section on engagement at Monchy of 10 April 1917. French and Haig entered the war optimistic about the utility of their cavalry, and were proven right.

In the immediate aftermath of the war, two more conflicts in the east also supported the continued utility of horse cavalry. The Russian Civil War and the Soviet Polish War. In both these conflicts massed cavalry proved crucial in covering the vast territory and broken and swampy terrain where the battles were fought. Yet as the Chieftain says above, rapid improvements in mechanical reliability and performance seemed to finally foretell the end of mounted combat. To a great extent this was true. Britain maintained no mounted combat units only one mounted cavalry division entering WWII, and the US only one as well, which was quickly reorganized. Germany still had a single vestigial cavalry brigade, which over the course of the war grew into several divisions, all serving on the Eastern Front and the Balkans, largely for anti-partisan operations. Poland had long been famous for her cavalry, and despite the overall insurmountable situation Poland found herself in, her cavalry performed at a high level. At the Battle of Mokra, a Polish cavalry brigade used their modern anti-tank weapons to decimate 4rth Panzer Division, and halt the German advance. There was even at least one last successful cavalry charge, [recounted here by the late BU history professor and Polish Cavalry veteran Kamis Dziewanowski.] (https://www.polishnews.com/last-great-charge-of-the-polish-cavalry/3)