Why didn't the French fleet at Mers-el-Kebir comply with the British demands?
In general, it was down to British misunderstandings about the motives and wants of the French, and therefore mishandling the negotiations.
On the 22nd June 1940, the French and German governments signed an armistice at Compiegne. This included two articles relevant to British interest in the French Navy. The first, Article 8, called for the French Navy to return to ports in Metropolitan France, where it would be disarmed under Italian and German supervision. It included guarantees that the Germans would not use the fleet for war. To the British, this was unacceptable; it would essentially place the French Navy under complete Axis control. Equally important, but somewhat overlooked by the British, was Article 10. This stated that anyone who left France to fight against Germany would be considered a franc-tireur (essentially, an person unprotected by the laws of war), and forbade the French government transferring any assets to the British. This was a powerful deterrent to anyone considering joining the British. The Germans had inflicted harsh reprisals on franc-tireurs during WWI, and it was clear that such policies would be repeated.
Over the next few days, the British tried to determine what the French position would be. The French, especially Admiral Darlan, assured them that the French Navy would not fall into German hands, and would be scuttled as soon as any such move would be taken. As far as the French were concerned, this was all that needed to be said. The British were unconvinced; as far as they were concerned, the fleet would be in German hands from the moment it entered a port in Metropolitan France. Such an option was unacceptable. With the French ships in German hands, the Germans would pose a major threat to Britain. The RN could not spare enough ships to counter the French, Italians and German fleet. As such, if necessary decisive action had to be taken to ensure that the fleet remained out of Axis hands. Planning for an attack on Mers-el-Kebir began on the 30th June, when Vice-Admiral Somerville (commanding Force H), held a commanders' meeting at Gibraltar. Meanwhile, the Cabinet and Admiralty were at work determining what steps should be taken, and if it was possible to avoid bloodshed.
On the next day, Somerville was presented with the list of British demands for the French fleet in North Africa. There were three options: the fleet could join the British, could bring their ships to British ports and be repatriated (the ships would not then be used against Germany or Italy, so the crews would not technically be breaking the armistice), or the fleet could be demilitarised by the British. If the French agreed to none of these, then Somerville would have to sink their ships. Several of his officers with experience with the French Navy, including Captain Holland of the Ark Royal (who had been Naval Attache in Paris), disagreed with the latter course of action, believing that it would alienate the French as a whole. As a result, Somerville made a number of alternative proposals to the Admiralty, hoping that the negotiating position would be softened.
On the 2nd, a new set of demands arrived. While the first and second options were the same, the third had changed. Rather than having the British demilitarise the ships then and there, the fleet could be taken to the West Indies and demilitarised there, or placed under American jurisdiction, The fleet could be demilitarised in North African ports, but only as a last resort to prevent bloodshed, and only if the option was brought up by the French. When communicating these to his officers, Somerville changed this last point somewhat, adding it as another point in the negotiating position.
On the 3rd, the fleet arrived off Mers-el-Kebir, and Captain Holland was dispatched aboard the destroyer Foxhound to serve as Somerville's emissary. This was the first faux pas made by the British; Admiral Gensoul, commanding the French force at Mers-el-Kebir, was reportedly offended by the low rank of the negotiating officer. Shortly after arriving, Foxhound broadcast, using her morse lamps, a public appeal to the fleet. This was another mistake; the appeal was poorly translated to French, and could be read by anyone in the fleet, not a good thing for private negotiations on such a sensitive topic. Further offence to Gensoul came from the fact that what was supposed to be a personal appeal from Somerville had not been signed (and thus could have come from anyone). Gensoul's negotiating position was what the French position had always been: the French fleet would not allow itself to fall into German hands, and would be scuttled if this seemed likely. Holland was generally sympathetic, but this was not an option for the British. Following this first round of negotiations, Somerville reported back to the Admiralty. The prospect of allowing the French fleet to disarm in place was mentioned, but was discarded for fear that it would weaken the British negotiating stance. However, Holland may have offered such a possibility on his own initiative. Meanwhile, Gensoul was communicating with Admiral Darlan. Darlan was unwilling to compromise. This was mainly a personal decision, made while preparing for a trip to Vichy to attend a meeting of the Council of Ministers.
Meanwhile, at Mers-el-Kebir, Somerville was beginning to worry that the French might escape the port. Information from Holland and from a scout aircraft suggested that preparations for an escape were in progress. As such, he prepared Force H for an engagement, and ordered Ark Royal to begin flying off minelaying aircraft to block the channel. As this was happening, he began to have second thoughts. Worried about the precise terms of the negotiation, with Holland insisting that further negotiations might have an effect, and with refined scouting reports suggesting that the French were not actually preparing to leave, he called off the bombardment. Unfortunately he was too late to prevent the minelaying aircraft dropping their mines. At 14:50, Gensoul requested further negotiations. Foxhound had slowly been edging away from the port, and so it took Holland over an hour to arrive at Gensoul's flagship. Gensoul was not happy; he did not like the way the British had negotiated - with the fleet offshore, it was essentially an ultimatum - and he was not happy about the minefield, which made accepting any of the first three terms impossible. He did, however, show Holland his instructions, which were that, should the Germans make any attempt to take ownership of the fleet, it was to proceed to the USA, or be scuttled if that was impractical. This was close to the British terms, and it might have been possible to work out a compromise. However, it was too late in the day for these. Somerville did not have time for further negotiations, as to do so would remove any chance of destroying the French fleet in the event they went bad. Somerville could not take more time, as his instructions from the Admiralty did not give him this latitude.