This is a question that needs two parts to the answer.

The first part is that question itself is confused.

Let us first set aside the first misconception: the Cromwell was not, in any way, an infantry tank. It was a successful, fast, well-armed cruiser tank of the late war (it first saw action on D Day) whose only failing was that owing to parallel development under the pressures of war had a turret too small to accommodate the large breech of the 77mm high-velocity gun. That forced it to operate in mixed units with Sherman "Firefly" tanks equipped with the 17lb AT gun. With a slightly enlarged turret to solve this problem it became the Comet, unarguably one of the best medium tanks of the war.

The question "why did the British invest in Matildas when the Sherman was so much better?" has a simple answer: they didn't. The Matilda II was specified in 1936 and developed from 1937 onwards, when tank doctrine was very much about World War 1 type wars (see below). It was already obsolescent by 1941, when it started being replaced by the faster Valentine which although nominally an "infantry" tank was derived from a cruiser tank chassis. By mid-1942 they were mostly all out of use, and there were only a handful still in service by the second Battle of El Alamein (October/November 1942). Which was, not coincidentally, the first combat use of the M4 Sherman, which had only started to be developed in 1940. This is essentially "why did Britain build all those Hurricanes when the P51D is so much better?" for tanks: the pace of development in military technology between 1936 and 1942 is simply incredible.

There were three "infantry tanks" which saw substantial service after the fall of France. The Matilda II was reasonably effective in the desert war until its armour was outpaced by developments in artillery, although its speed was an issue. The only other tank available in a similar role prior to the Sherman was the M3 Lee/Grant, which again did not start being developed until 1940 and but was available by late 1941.

In any event the Matilda had started to be replaced by the Valentine, which was faster (being derived from a cruiser tank) but, like the Matilda, under-gunned. It was developed in a hurry from 1940 onwards, but on an existing platform. The Matilda was out of production by 1943 but the final examples did not serve in theatres where they had to face German tanks, other than the chassis being sometimes used for special-purpose vehicles.

And finally there was the Churchill, which served with some distinction throughout the war as, in essence, a heavy tank. Aside from its chassis being the basis for a wide range of specialised vehicles, the later variants were effective in France in late 1944 -- ironically, in some way precisely because they were designed for mobility on difficult French terrain.

So the first part of the question boils down to "why did the British have the Matilda II in Africa in 1941, rather than tanks that did not at that point exist?" I think the answer to that is self-evident: it was the only tank available.

As to _why_ the Matilda II was slow, heavy and under-gunned, the answer is as the OP suggests, the mistaken "infantry tank" doctrine. Inter-war Britain was still planning for a static war of attrition in the manner of the first world war. But apart from Germany, pretty well everyone's tank doctrine circa 1936 turned out to be wrong. Most designs of 1936 were, with hindsight, simply too light in both armour and armament, and the British infantry tanks were an aberration that although slow were at least sufficiently armoured even if they were not well armed. The American comparator for the Matilda II is not the (later) M3 or the (later) M4 or whatever, it is the American M1 light tank. In that context, the Matilda II looks rather good.

Tank doctrine is the thing that Germany, under Guderian, got right. Once it became apparent which way the wind was blowing, everyone followed, and the second world war for the Allies is mostly a story of developing new, good, effective medium tanks (the T34, the M4 and the Cromwell/Comet) while getting as much service as possible out of the design and tooling of such of the earlier designs as could be stretched out.

Introducing new tanks into service is hard, as the Germans found. It requires large inventories of parts, training both for crews and maintenance units, detail production and reliability engineering, etc. The wartime German designs (Panther and Tiger, largely) were immensely handicapped by this, and never achieved full operational effectiveness. As I have said in other contexts, had the Germans been operating a boutique weapons design consultancy for the cold war to come they would have done very well, but in practical terms it is an open question whether the resources expended on the Panther and Tiger would have been used to produce Panzer IVs. The pragmatic decision by the British was to use the tanks they had, build the tanks they could build, buy the tanks they could buy and make the best of it. Yes, almost certainly, in 1936 German doctrine was right and British doctrine was wrong. But the British pragmatically adopted the aspects of modern tank doctrine that they could: what else could they do?

But why did they keep building and designing new bulky, slow infantry tanks, rather than investing earlier in a medium tank with higher speed and bigger guns?

It's easy to say that the British went down the wrong path in hindsight, but the development of British armour doctrine is fairly reasonable when you look at it in context of its time.

Let's go back to the early 1920s. The British adopted first the Medium Tank Mk.I then the Medium Tank Mk.II. These tanks could reach a speed of 24 kph, which was more than twice as fast as Renault FT light tanks, and had a 47 mm gun that could penetrate up to 30 mm of armour, enough to defeat any enemy tank at the time. There was also a close support variant with a 3.7" howitzer. Here we have a high speed medium tank with a big gun!

The family of tanks continued into the Medium Tank Mk.III, but only three prototypes were built. This tank was supposed to be able to reach speeds of 41.5 kph with a more powerful engine, but a blow was struck to the British armoured force with the disbanding of the Experimental Mechanized Force and cuts to the budget of land forces in 1927. The Mk.III died from this blow, and the medium tank concept with it. This was a very bad move for the British, since the Mk.III inspired the Soviet T-28, arguably the best interbellum medium tank.

Experimental work continued and it quickly turned out that you can't have everything in one tank. In 1934 the army requested a tank with the capabilities of the Medium Tank Mk.III, but lighter (12 tons instead of 16). At the same time, the army was eying the French Renault NC, so they also asked for something similar: a thickly armoured tank armed only with a machine gun.

These two directions evolved further in the mid-1930s. The British were impressed by the Soviet BT tanks at the Great Kiev Exercise and wanted something like that for themselves, and so the cruiser tanks evolved to be more like the BT (bulletproof armour, Christie suspension, high speed). At the same time, requirements for armour grew too, so you see an increase in armour on both the Cruisers (Cruiser Tank Mk.IV) and the infantry tanks (Infantry Tank Mk.II) although the latter obviously got the brunt of the armour upgrades. The armament was also modernized with the high velocity 40 mm 2-pounder and 3" howtizer for close support tanks.

Looking at the tank armament at the start of WWII, one would hesitate to state the the British bet wrong. Their "medium" tank was superior to the German Pz.Kpfw.III. The armament was the same (remember that there were no 50 mm guns yet), the armour of the Cruiser Tank Mk.IV was thicker (early Panzer IIIs had 14 mm of armour, the same as the Cruiser. Tank Mk.III), the Cruiser tank was faster. The infantry tanks had superior armour that withstood anything up to German AA guns.

It made sense for the next generation of British tanks to look exactly the same. The Cruiser Tanks Mk.V and Mk.VI departed little from the Mk.IV concept and the Infantry Tank Mk.IV was still a slow and thickly armoured tank (although it had a howitzer in the hull, a feature that was quickly dropped).

The problems with this concept were only realized in the battles in North Africa. The long sight lines meant that fighting was happening at much longer ranges than in France. In this case the smaller and (slightly) faster (but more importantly, much cheaper) Valentine proved superior to the cumbersome Matilda. Also it had superior reliability and no issues with long range driving that plagued the Matilda. The new Churchill also turned out to be pretty okay against 88 mm guns at long range.

The Cruisers weren't in such good shape though. They could avoid 88 mm gun fire at long ranges due to their speed, but once they had to engage German tanks at closer ranges they couldn't compete. The solution was the "heavy cruiser" concept that surfaced at the end of 1941: a tank with the armour of an infantry tank and the speed of a cruiser tank. This was your "high speed medium tank with a bigger gun", as it would have the 57 mm 6-pounder gun that allowed it to fight German tanks at long ranges.

The problem with the heavy cruiser is that there were many ideas about how to build it. Development split into three paths: the A24 Cavalier built using Cruiser Tank Mk.VI components, and then two brand new tanks: the A27M Cromwell (using a new Meteor engine) and A27L Centaur (using the old Liberty engine). The plan was to put these tanks into service in 1942, but it took far longer than expected to iron out all the bugs. By the time the tanks saw battle in 1944, it was clear that the concept was already obsolete. The replacements were already in development: a minor upgrade to the Cromwell concept (Comet) that saw battle at the tail end of the war in Europe, and a major redesign that abandoned every tradition in British tank building (the Centurion).The former was just a stopgap, but the latter resulted in a fantastic tank series that served for decades after the war.

As for the Churchill, while there were competing tank designs that were supposed to replace it, there were no proposals to do away with the concept of an infantry tank in general. If anything, the Churchills only got slower and more thickly armoured as the war went on. Like heavy tanks in other nations, they were still relatively specialized vehicles, the main type of tank in the British army was the heavy cruiser, whether the Sherman or the Cromwell.

​

Sources and further reading:

http://www.tankarchives.ca/2016/03/medium-tank-mki-first-of-maneuver-tanks.html

http://www.tankarchives.ca/2016/12/medium-tank-mkii-interbellum-long-liver.html

http://www.tankarchives.ca/2017/01/medium-tank-mkiii-britains-cerberus.html

http://www.tankarchives.ca/2019/01/the-first-cruiser.html

http://www.tankarchives.ca/2019/01/cruiser-tank-mkii-with-best-intentions.html

http://www.tankarchives.ca/2017/04/infantry-tank-mki-first-infantry-tank.html

http://www.tankarchives.ca/2018/05/short-term-queen-of-desert.html

http://www.tankarchives.ca/2016/01/britains-christie.html

http://www.tankarchives.ca/2016/05/cruiser-iv-bit-more-armour.html

https://warspot.net/246-first-among-equals

https://warspot.net/382-front-line-prime-minister

https://warspot.ru/15292-bednye-rodstvenniki-kromvelya

https://warspot.ru/13466-kromvel-luchshe-pozdno-chem-nikogda