There's not really any brutalist buildings around my location, but I absolutely love them. They are some of the most interesting building, both in terms of visual appearance and the thoughts that has gone into their design. To me they don't instill horror, but a sense of calm and safety.
What sometimes happens when people say they dislike brutalism, and what is does to people living in the buildings, is they focus on the architecture and not the horrible property mismanagement. The UK has a number of hated brutalist towers and the misery of those living in the building are ascribed to the architecture, not the fact that the buildings are not properly maintained, or that the cities stuffed the flats with the people who are incredibly poor, addicts, in need of mental care, education, support or a mix of all of those things. Now it is also true that many of these buildings are old, typically from the 60s and 70s, and their design no longer suites modern living, but that's true almost all types of architecture. A 1950s brick house barely fits a modern family.
The problem might be exactly that: A brutalist building doesn't lie. If you don't take care of it, and its surroundings, the building will let you know. Nothing is hidden, all of your societal problems are on full public display with a brutalist building.