As I basically understand it, the Black Legend formed as the rest of Europe learned about how oppressive the Spanish were to indigenous peoples. Why is it considered to be an unfair representation of Spanish colonialism when objectively speaking the Spanish were very oppressive to indigenous peoples?
Here's three things:
Most of the perpetrators of the Black Legend weren't simply saying "the Spanish are very oppressive to indigenous peoples." The Pope called them "the scum of the Earth." Immanuel Kant, in Observations on the Feeling of the Beautiful and Sublime, calls the Spaniard cruel, centuries behind in science, lazy, ignorant of reform, and un-European (derogatively). These are not objective utterances. Pope Paul IV, for instance, had what you might call a personal vendetta against the Spanish, having hated growing up there and being a political opponent of its royalty. Black Legend wasn't always a criticism of deeds, but the defaming of a people.
In modern popular accounts, criticism of the Spanish is typically tied with a hidden "noble savage" motif. The narrative is as follows: the Inca are living their happy peaceful life in the Andes. They've got a lot of cool things: roads with regular storehouses/rest stops, a diverse religious culture, nifty stoneworking techniques, etc. Then the white man comes with his horses and guns and tramples over them, forcing them to near slavery in tin and silver mines. Poor Inca! That was rightfully their land, and they never did anything to deserve that! What's wrong with this? Any time you see this story, the Inca are the good guys and the Spanish are bad. As targustargus tried to point out, this completely lacks nuance and knowledge of Andean prehistory. For instance, the Inca were only native to the Cuzco area. They had conquered the rest of the Andes within the past hundred years, and they were still conquering places in the far north when the Spanish first arrived. Like the Spanish, they were a small ethnic group with big ambitions for conquest who laid their own culture and infrastructures on top of subjugated groups and who appropriated resources, culture, and religion for their own goals. Additionally, the mit'a system you bring up was in fact an effective method of creating an infrastructure. In the Inca empire's brief existence though, this was often an infrastructure for conquest, with roads to facilitate troop movement and storehouses to extend campaigns. If a people joined the empire through treaty, then such a taxation might be reasonable. But if you're a former citizen of the enormous city of Chan Chan, defeated by Topa Inca Yupanqui's army ~1470, being forced to help replicate your own destruction is unspeakably humiliating. As an archaeologist of pre-Inca cultures, I am regularly frustrated by how much cultural knowledge has been lost due to the Inca conquest.
In addition, popular accounts focus on the Spanish actions and downplay native agency. It's always "The Spanish conquered the Inca" and it's never "The Inca attempted political intrigue and military outmanuevering against the Spanish, but failed." The conquest changed the Spanish as it did the Andeans.
TL;DR Popular narratives downplay any benefits of the Spanish conquest and romanticize the prior Inca conquerors.
Some good sources for complex looks at the conquest period:
Gose, P. (2008). Invaders as ancestors: on the intercultural making and unmaking of Spanish colonialism in the Andes. Toronto; Buffalo: University of Toronto Press.
Mumford, J. R. (2012). Vertical empire [electronic resource]: the general resettlement of Indians in the colonial Andes. Durham [N.C.]: Duke University Press.
Ramos, G. (2010). Death and conversion in the Andes: Lima and Cuzco, 1532-1670. Notre Dame, Ind.: University of Notre Dame Press.
Wernke, S. A. (2013). Negotiated settlements : Andean communities and landscapes under Inka and Spanish colonialism. Gainesville: University Press of Florida.
CommodoreCoCo has brought up some good point, but one thing I'd add is that the Black Legend is not about the brutality of European conquerors in the New World in general, but claims the Spanish were uniquely bad.
The Spanish did some bad things, but were they so much worse than the Dutch? The English? The French? The Portuguese?
In some ways, I would say they took different approaches. For example, the English were more likely to regard American Indians as being foreign states, while the Spanish were more likely to regard them as being Spanish subjects. Although thanks to the power differences, bigotry, etc, the Spanish authorities treated their Indian subjects worse than they treated, say, the typical Castilian peasant, and the English treated the Indian foreigners with more cruelty in war and less respect in peace than they did with, say, the Dutch when fighting the Anglo-Dutch wars or the treaties ending them. (This isn't to say that, say, the English never engaged in outright conquest, or that the Spanish never recognized the sovereignty of American Indian powers or signed treaties with them; just that the general patterns varied to some degree.)
Can we say the Spanish were worse though? That's a tougher call. From a sort of "international law" perspective, the Spanish approach of outright, often unprovoked conquest seems worse, but bringing people under the authority of the crown did give them some rudimentary sense of responsibility, whereas, say, the English coercing an Indian tribe into signing a "cede your land to us, and move somewhere else and starve to death" treaty felt much less responsibility.