I'm afraid I don't have time to give a more detailed answer, but Brutalism suffered from a) that name, which as another poster stated, doesn't have the meaning that such a term connotes in English; b) a sustained campaign against it by conservatives (notably Prince Charles in the UK) and Postmodernists and, following this, c) a long period of neglect which allowed many of the buildings to become eyesores.

It's worth noting that like many forms of Modernism, this style may seem ugly to begin with (as with some Modernist Literature or music particularly) and so requires you to do some work in order to get it. An idea such a truth to materials is simply something that the general public don't care about, and it's not until you consider this that the unfinished look of the concrete used starts to make sense. The 'make it new' aspect of it, as well as its utopianism (dystopianism if you don't like it, I guess) I think led to it being used in places that wanted to suggest a break with the past and a newfound modernity. It's also worth noting that it's a style that simply could not have existed without the development of new building materials in the C20th, which is also an attractive feature if you are interested in it as a Modernist form.

There has been over the past few years a reappraisal of the style led by Owen Hatherley in the UK which has led to many of these buildings becoming listed and redeveloped. Here's a brief newspaper article which discusses its aesthetics.

More:

Calder, Barnabus, Raw Concrete

Hatherly, Owen, Militant Modernism, 'Strange, Angry Objects' LRB November 2016 (and the books he cites within).

Reddit Sub.

edit: all kinds of stuff

An evaluation of any architectural form - like any art form - is subjective, so "fair" is probably not the right word. An assessment that a type of architecture is ugly or beautiful is in the subjective eye of the beholder, and if one decade's assessment leans in one or the other direction, all one need do is to wait a while and the tendency to view it another way will likely become in vogue.

Part of the problem in evaluating Brutalism is that as James Stevens Curl, Oxford Dictionary of Architecture and Landscape Architecture (second edition 2006) indicates, the term Brutalism is "so loosely used it could mean anything intended by those employing it." In addition, architectural historians sometimes make a distinction between the original Brutalism and "New Brutalism", which was particularly common in Britain, inspired by the original French designs by Le Corbusier, who worked with this style in the late 1940s to the early 1950s.

The British sequel (and then its American epilogue) to the style won particular scorn while also clouding the origin of the name. The term was apparently originally derived from the French term "brut", referring to the unfinished concrete forms. In Britain, the term gained some energy because its lead practitioner Peter Smithson had a brooding brow that reminded colleagues of a bust of Brutus, winning him that as a nickname. His enthusiasm for the French style confused the etymology in English.

In short, it is not fair to say that Brutalism is unapologetically ugly since that is a subjective evaluation. That said, anyone who says it is anything but ugly as hell, is clearly wrong. At least in my opinion.

There is always more to be said, but you may be interested in the answers given to these past questions:

What are the roots of this distinctively monumental style of Soviet architecture?, answered by /u/andriah_blashkovich

Was the architecture built in the Soviet Union and the Eastern Bloc built in a grim style intentionally?, answered by /u/kieslowskifan

I'm going through Le Corbusier's Vers une architecture (1923) right now. I want to introduce a paragraph that draws attention to the source the mindset of the era. My copy of the book is a Hungarian translation written in 1981 from a 1966 French reprint. I'm sorry if something gets messy with all these translations.

The architects and artists of the avantgarde unevitably witnessed the engineered wonders of the era. They saw the birth and spread of steamboats, then airplanes, then automobiles, but also machines for made for harvesting. These people saw them in a new light, different from other people. The eyes of architects or artists saw and explored the new methods of manufacturing. The new purposes which these machines created, or the old they reinvented. These people also saw the ruling architectural styles of the era. Organic and overly artistic Art nouveau co-existed with a more rigid, but eclectic Classicism. People were debating whether to go for form, function in constructs. If one was more important than the other, if they can be balanced. In this time period architects and artists noticed the similary between "houses" and "machines". More so, since buildings too are a product of engineering and their raising involves measuring, planning, plotting, managing resources and they also produce something in the end. Just like machines have products. Houses produce human life. Such or similar thought processes have led Le Corbusier to the conclusion that a house is a machine for living in.

With such an industrious mindset it's easier to see how function was reinvented, how form was repurposed. The crisis of post-war housing also allowed for new architecture. This is how l'Unité d'habitation (1952) was planned and built. Le Corbusier looked at houses as one unit for living in. He stacked them on top of each other. The modable floorplan was fitted together, a facade was built in front of them. People could walk out on the top, or under the house which stood on stilts. These european nations being rich meant that they had the resources to build such constructs, but they also had the necessity because of the rising population. How different political leaderships adapted concrete as resource to housing masses is a bit different question. l'Unité d'habitation was built in Marseilles, Nantes-Rezé, Berlin, Briey-en-Forêt and Firminy. A very similar apartment complex in Hungary was never built because it was from Western influence. Then under Communist regime we got quite many brutal looking, concrete flat complexes but these weren't Western in production or ideology as much. As a personal opinion on the matter I wouldn't call béton brut ugly. There are buildings that are ugly and there are others that are not.

Please note: despite the subquestion asking if it is fair to describe brutalist architecture as ugly, comments that do not engage with the historical question of how the style rose to prominence will be removed. We require that answers be comprehensive and in-depth and about history.