Whilst the British Empire certainly takes center stage in many contemporary discussions regarding the legacy of colonialism and the impact it has left on various parts of the world, take note that by no means is the discussion on Empire and its consequences exclusively a British notion. In the timeframe OP mentioned in their question, the impacts of Spanish and Portuguese colonialism in South America remain a key topic of discussion for the historical academia in those topics. Further still, we can find plenty of works of literature on the impact of the Dutch with their colonies in the East Indies (modern day Indonesia), their brief spell on the Cape of Africa, and various other colonial outposts in the West Indies. Even that longstanding rival of the British in colonial ventures, the French Empire, remains a highly studied area of imperial history. In short, British colonialism is by no means the only focus of discussions on modern colonialism, but it does take up a large portion of the discourse because of how much of a reach the British Empire possessed at its zenith.

Is this because Britain's actions are the most commonly known?

Because of the very size of the British Empire, its legacy after the post-colonial period is one of if not the most well-known topics in late modern imperial history. Again however, this all depends on which discussions and areas of imperial historiography you decide to investigate/read. It is no surprise that perhaps the most well-known and heavily-written about sub-topic of imperial legacy: the Indian experience of colonialism, is linked directly and almost exclusively to Britain; after all, for almost a century before the founding of the modern Republic of India, the subcontinent was commonly referred to in official legislature and diplomatic correspondence as the British Raj. A considerable portion of Africa as well, has had to deal with the scars of a colonial history where the British were key stakeholders, enforcing their will (though not necessarily way of life) across many pre-colonial civilisations and societies for decades. The system of apartheid in South Africa (which resonates in the ongoing racism problems there to this day) was largely a result of the colonial system introduced by British (and going back even further, though to a lesser extent, the Dutch). It is perhaps no coincidence that in many school curriculums today, the British Empire is a key area of study in modern history, because more often than not that curriculum is practiced in a nation or region where the British had some presence at some point in their imperial past.

Or is it that, despite other countries getting as involved in the negative actions Britain did, Britain's actions are worse? Or maybe Britain was more "successful"?

All these questions might merit their own separate posts, as other contributors could (and should) weigh in on the litany of atrocities and heinous actions undertaken in the process of expanding various other empires throughout history. I would however, caution against merely viewing this issue as a numbers game, where entire arguments are based on the assertion that "This empire killed X amount of people whilst this empire only killed Y amount of people". Every empire has been guilty of "negative actions", and there are those in the historical academia who argue that such a process is an inevitability in the process of empire-building.

That the British Empire was more successful is certainly, at least on the surface, a possible explanation for why it dominates the post-colonial narrative of so many countries. This links back to the simple fact that London reigned over huge swathes of territory, as well as having their hand in many more nation states (the so-called "zones of influence"). However, by no means was the British Empire always successful or indeed, always an Empire. There was a time when the Tudor and Stuart monarchs of Britain envied the Spanish and Portuguese kings and queens for the sheer haul of treasure that their colonies in Latin and South America (the "New World" as it was called back then) yielded each year. Likewise, the British have suffered more than their fair share of setbacks, humiliations, and disappointments on the road to Empire. Militarily, for every victory such that at Plassey in India, there was a defeat such as that at Islandlwana in Zululand.

In sum then, whilst the memory of the British Empire is in the relative sense perhaps the most dominating one due to its historical influence and character, it by no means is the absolute "king" (excuse the pun) of imperial history and I reckon you would not need to search hard enough in a library, journal, or even subreddit (such as AH) to find a plethora of writings on the other empires which have left their mark on our world. More can always be said on this interesting question, but I hope that this overview of the matter suffices as well (though feel free to ask follow-ups as well!). To end this response, the following quote from imperial historian John Darwin might be of some use:

"We live in a world that empires have made. Indeed, most of the modern world is the relic of empires: colonial and pre-colonial, African, Asian, European, and American. Its history and culture is riddled with the memories, aspirations, institutions, and grievances left behind by those empires. The largest if not grandest of these was the empire laboriously assembled by the British across more than three centuries. No less than one quarter of today's sovereign states were hewn from its fabric. For that reason alone, its impact was second to none."

Further Reading

This trilogy of books ought to help get you well-situated on the reality of the British Empire, and of course they all paint a comprehensive picture on the sheer breadth and depth of its legacy in the modern age.

• Ashley Jackson, "The British Empire: A Very Short Introduction"
• John Darwin, "The Empire Project: The Rise and Fall of the British World System, 1830-1970"
• John Darwin, "Unfinished Empire: The Global Expansion of Britain"