This submission has been removed because it violates the rule on poll-type questions. These poll-type questions do not lend themselves to answers with a firm foundation in sources and research, and the resulting threads usually turn into monsters with enormous speculation and little focussed discussion. “Most”, “least”, "best" and "worst" questions usually lead to vague, subjective, and speculative answers.
This is a very hard question to answer, since 'impact' and 'human rights violations' are not very quantifiable, and 'colonial power' is quite hard to define.
All the colonial powers were responsible for their fair share of bloodbaths though. The German Empire for instance oversaw the Herero and Nama Genocides in Namibia in 1904-1907, where more than 3/4 of the original Herero population of 80,000 were killed. If you count Leopold II (as opposed to the Belgian Empire) as a colonial power, he was responsible for up to 10 million deaths in the Congo Free State. The Belgian government itself was also responsible for a large number of atrocities in what is now the DRC and Rwanda, as well as their generally despotic rule.
In terms of post/neo-colonialism, however, I'd say that France was by far the worst power, through their policy of Francafrique that overthrew regimes who went against French interests, encouraged the Rwandan genocide, and still continues to exert malign influence in Gabon.