First off, some previous answers that are relevant to this subject, written by yours truly: Why did Britain not assist the US in Vietnam and did the US use British tactics that was used in the Malayan Emergency (if so why wasn't it successful?), Why was the American Intervention in Vietnam such a failure in comparison to British Anti-Communist action in Malaya?.

I'd like to point out that your observation of the nature of the Vietnam War is actually very true; the Vietnam War was not waged as a counterinsurgency by the US and thus can't really be called a counterinsurgency conflict (even though it.. Sort of was). At the same time, they were facing guerrillas and while the NVA was a conventional force, they did fight as asymmetrical insurgents and only engaged the US conventionally a few times as opposed to the more focused guerilla action. On the other hand, the VC were equipped as guerrillas yet in '65-'66 fought conventional battles. So, the nature of the conflict (i.e. the stress applied to ordinary US soldiers etc.) was very much that of counterinsurgency but it was not waged as one.

Britain never approached the Malayan emergency as a conventional conflict

No, this isn't true at all. In fact, the initial strategy in Malaya was more based on colonial warfare than counterinsurgency. That is, large sweeps of search and destroy patrols and a very heavy-handed approach to the indigenous population. It is perhaps telling that the few atrocities that did occur in the conflict happened before any sort of strategy of counterinsurgency was set into motion.

Britain successfully waged a PR campaign; key to COIN. Is this true? Given the small-scale of the Malayan insurgency, was hearts and minds necessary? Was the Malayan insurgency ever at serious risk of expansion?

Hears and minds was indeed necessary. Yes, the Malayan Emergency was indeed a small-case affair. There was no real danger that it would expand beyond Malaya or into the major Malaya population. However, the Chinese minority had to be won over and to prevent more violence, hearts and mind was the way to go. The early years of the conflict had proven that a more ruthless approach had not given any results. Hearts and minds not only won over the population - it also meant that there would be an abundance of intelligence which was used to track down and kill insurgents and capture their supplies.

Finally, is it even proper to compare these two conflicts?

It's important and interesting to contrast these conflicts, to see what was done wrong and right. However, one has to realize that there isn't a "one size fits all" counterinsurgency strategy that is going to work in every given conflict. Each and every conflict, whether it be Vietnam, Malaya, Algeria, Aden, Afghanistan and so on, has its own particular set of local and unique conditions which had to be dealt with. Malaya, for example, was lucky to be experiencing a financial boom during the Emergency which made it easier to finance the effort going into fighting the insurgents but also for the infrastructure and overall financial well-being of the country.

One can certainly draw great lessons from Malaya: A good focus on hearts and minds to win over population is an excellent approach when carried out by individuals who have knowledge about the culture and language. Separation of population is a classical counterinsurgency strategy and same thing applies to the Strategic Hamlets. A good COIN strategy is deceivingly simple: more civic action, less shooting. A reliance on smaller units of armed forces fighting guerrillas together with strong intelligence gained from the local population usually leads to great successes. To co-ordinate political, economical and military agencies and leadership into one is also vital, so everyone are pulling towards the same goal.

I think you would find Dr. John A. Nagl's Learning to Eat Soup with a Knife: Counterinsurgency Lessons from Malaya and Vietnam (2005, University of Chicago Press) to be very enlightening in understanding why the British did as they did in Malaya as opposed to the United States in Vietnam.