Anecdote: For my programming classes, one example I use is a simple browser. It doesn't do CSS or Javacript, so display is primitive, but it works. On some sites. Many sites, especially the big ones, see that it's an unknown browser, and refuse to send content. Probably they think it's a bot. But even if it were, what's wrong with bots, as long as they're well-behaved? What kind of closed web have we let the megacorp…
Playing the devil's advocate: Why shouldn't a server get to decide which clients it wants to talk to?
But each person developing a web scrapping bot realizes at most after a week that being honest with User-agent has negative impact on how well it works, and changing it to existing browser takes literally seconds.