I asked about this in the Southern Africa AMA six months ago.

/u/profrhodes gave a pretty comprehensive answer here.

For those on mobile etc, here's what they said:

The question of foreign military intervention in Rhodesia was suggested from 1963 all the way up to 1979 by numerous international groups, and especially the independent African states. However, from as early as October, 1965, Harold Wilson (the British PM) made clear that no British government would be prepared to use force in Rhodesia, unless Rhodesia invaded another country.

The reasons behind the decision to throw away ‘the most powerful bargaining counter available to the British government in its dealings with the Smith regime’ are complex and have already been examined in several articles, notably a 1975 essay from Douglas G. Anglin, and more recent academic work from Elaine Windrich and Philip Murphy, all three of which assess the realities of the British military intervening in Rhodesia. The arguments put forward as to why military action against Rhodesia would have been very, very difficult make use of two points.

First, that ‘all the evidence before us was to the effect that [Rhodesia’s] forces were well-armed and well-trained; and they would fight. This would not be a colonial expedition but a medium-sized war of uncertain duration.’ The Rhodesian army was one of the most professional military forces on the continent, and, at least in 1965, had the largest air force after Egypt. Although it was a relatively small standing army, it was well trained with many veterans of WW2 and had equipment from Britain, including fighter jets. Military intervention would not be as simple as dropping some soldiers in to put down the white government.

Secondly, Denis Healey, Secretary of State for Defence (1964-1970), emphasised that ‘the British armed forces could not be trusted to execute orders for a military intervention against their Rhodesian “kith and kin.”’ Would British soldiers willing shoot at white British soldiers wearing Rhodesian uniforms? Many Rhodesians had been born in Britain and emigrated to Rhodesia after 1945, and for many British people Rhodesia was seen as the last-stand of the British empire in Africa. Could the British Army have trusted their soldiers to fight against their own brothers? There were reports given to the British government that military leaders had said they would directly refuse any orders given which involved firing upon Rhodesian whites.

The international community gave Britain an ultimatum on many occasions along the lines of 'you deal with this, or we will' which as I mentioned in another answer fed into Britain's fear of the USSR gaining a foothold on the continent under the pretence of a peacekeeping mission (Red Army in Blue Berets). However, successive British governments insisted Rhodesia was their problem to solve and they would not consider force and nobody else could either!

Thanks /u/transitiverelation for passing that answer along! I'll expand upon it a little bit as that was a fairly brief summary of a very complex topic. /u/i8ontario is completely right that international support for military intervention was very, very strong in the immediate aftermath of UDI. Most of the African states, plus Russia, China and most non-colonial nations were explicit in demanding military action within the UN. Why Britain didn't invade therefore needs explaining.

First, it is crucial to note that neither the British government nor the British public were unanimously of the opinion that force should be ruled out. Opinion was actually very divided for a variety of reasons, but a lot of the reasons that were made public belied other factors. When Harold Wilson, the British PM, announced to a Rhodesian press conference on 30 October, 1965 that

If there are those in this country who are thinking in terms of a thunderbolt, hurtling through the sky and destroying the enemy, a thunderbolt in the shape of the Royal Air Force, let me say that his thunderbolt will not be coming, and to continue in this delusion wastes valuable time and misdirects valuable energies (Elaine Windrich, Britain and the Politics of Rhodesian Independence (New York, 1978), p.49)

he was almost certainly refuting the statement another British minister, Arthur Bottomley (Secretary of State for Commonwealth Relations, 1964-66), had made a month earlier when Bottomley had declared that British military intervention would being considered if either the Rhodesian Governor (Sir Humphrey Gibbs) or the Zambian government called for it. The reasons why Wilson did not want military intervention in Rhodesia in case of UDI were many, and include the reasons my quoted answer provides. First, that the Rhodesian military would be able to drag a war out indefinitely, and secondly, that the British troops would be unwilling to fight Rhodesian forces.

Both of these fears were to an extent justified. The Rhodesian military was certainly strong enough to make any British-led invasion a costly (both in terms of lives and money) and protracted war. There have been suggestions that in the case of a British military intervention, the Africans would likely have provided enough support to the invasion force to render the Rhodesian state impotent, but such beliefs were unfounded and rare amongst the British decision makers. The second argument, that the British troops might hesitate to fire upon Rhodesian 'kith and kin' had a little more support. When the British were still considering military action in 1965, air photographs of dropping zones and other information related to 'somewhere in Africa' were issued to D Squadron of the SAS (at the time in Aden) who would likely be the first troops into Rhodesia.

This led to a secret poll among the SAS ordinary ranks on the question: ‘If the order to go to Rhodesia is given, would you be willing to fight?’ Most men were not, and, even though they understood they could not refuse a direct command, they indicated they would seek to avoid a confrontation with the Rhodesians. It is believed that this had a ripple effect in British ranks in Aden and contributed to Wilson’s final decision to rule out the use of force. Although it is certain that any British reluctance to fire on their Rhodesian brothers-in-arms would have dissipated with the first British casualty, such a factor would still have been taken into consideration. (J.R.T. Wood, 'So Far and No Further', p.425)

Andrew Skeen, the last High Commissioner in Rhodesia and a military officer in the British army, also suggested that most British commanders would refuse to fight and that some had even declared they would join the Rhodesians in protecting the last bastion of white Britain in Africa. Dennis Healey (Minister of Defence) also emphasised that leaving aside the psychological problems of British troops fighting Rhodesian troops, there were not only the logistical difficulties of a British or UN intervention in Rhodesia, but also the danger of provoking Rhodesia into launching pre-emptive strikes against Zambia or sabotaging the Kariba hydroelectric station (on the Zambian border) and thereby shutting down Zambia's copper production. However, Joseph Grimmond (leader of the opposition Liberal Party) and Edward Heath (leader of the Conservatives) were both supportive of military action, until UDI was declared when they both agreed to focus on economic and political sanctions.

Secondly, Wilson and the British government argued that the British public would not have supported military action. Carl Watts' article ‘Killing Kith and Kin' (see footnote) asserted that the British public was both ill-informed on the issue of the military intervention and the Rhodesian crisis (and would therefore not understand the necessity of military action), and also that there was a vocal minority (about 22% of the British public) who explicitly supported the use of force against an independent Rhodesia. My own research into the issue of the British press, British public opinion and military action against Rhodesia has suggested that on the basis of a statistical analysis of the quantity of news articles regarding the use of armed force against Rhodesia, the prominent positioning of articles within the newspaper, and the relative ‘attractiveness’ of the articles’ headlines, the issue of military action against Rhodesia was undoubtedly a salient issue in the public agenda in the immediate aftermath of UDI. My research has also concluded that public opinion was certainly not unanimous in the aftermath of UDI, with a pro-force minority making justifiable arguments against an anti-force majority for British military intervention with the purpose of forestalling an invasion of Rhodesia from a foreign power. Watts' figure of 22% of the British public in support of military action, working off Gallup Polls from 1964 and 1965, and my research with newspapers from the period of UDI has affirmed that figure to be roughly accurate. In the immediate aftermath of UDI, it is justified to state that around three-quarters of the public were opposed to any military action, and often even to Wilson’s desire to crush the Smith regime. In response to an editorial alleging the entire British population was behind Wilson’s punitive measures against Rhodesia, one sardonic letter written to the Daily Mail stated quite simply that ‘the claim that a great measure of unity in confronting Rhodesia exists in Britain may turn out to be the overstatement of the year.’ (Implying that the unity came from support for Rhodesia).

There were some public voices, however, which went further and made clear the negative repercussions military action would have for Wilson’s Labour Party, as was emphasised in unambiguous terms in one blunt letter to the Daily Mail;

if Mr. Wilson considers – or allows – force to be used in Rhodesia he will have to be prepared for a large number of Smethwicks.

The town of Smethwick saw race riots in 1962 and the subsequent unseating of a Labour MP after apparent preferential treatment given to immigrant ethnic-minority parts of the population. The unofficial campaign slogan put about by the protesters was ‘if you want a nigger for a neighbour, vote Labour.’ Certainly, Wilson was aware that the majority of the British public would be uncomfortable about any use of force against those white Rhodesians who were 'simply protecting their lifestyle from the savage Africans.'

Finally, most of these factors come from the period immediately after UDI. So why didn't Britain invade later on? Well, simply put, it was because from around 1965 onwards the white Rhodesians found themselves engaged in a civil war with African nationalist forces. Britain had no intention of getting involved in the Liberation War to any extent, preferring instead to focus on the economic and political sanctions it had implemented. The only official British military presence in Rhodesia came in 1979 when a token peacekeeping British force was dispatched to ensure a peaceful transition from minority to majority rule. Rhodesia remained an embarrassment to the broader policy of decolonization and subsequently Britain preferred for the issue to be resolved without the use of force. The belief put forward by the British government during the 1960s, and 70s was that the Rhodesian Front government of Ian Smith would die a natural death at the hands of the African nationalist forces and the economic sanctions.

Anyhow, the tl;dr of why Britain did not invade is that the British government, press, and public were all relatively opposed to military force, and despite a vocal minority calling for war, the logistical problems, the strength of the Rhodesian military, and the belief that both British soldiers and the British public would not support any military action saw Britain to remain content with economic and political sanctions.

For those interested here are the references for the quotes from that earlier answer, plus a very short list of suggested further reading. If anybody has more questions, or wants some expansion on a specific topic, let me know!

the most powerful bargaining counter...

is a quote from Philip Murphy's article ‘“An Intricate and Distasteful Subject”: British Planning for the Use of Force Against the European Settlers of Central Africa, 1952-1965’, English Historical Review, Vol. 121, No.492 (Spring, 2006), pp.746-777, p.746

all the evidence before us...

is a quote from Sir Michael Stewart, Secretary of State for Foreign Affairs (1965-66), autobiography Life and Labour, p.169.

The Dennis Healey quote about Kith and Kin is from his autobiography The Time of My Life, p.332