This is a problem mainly for people who use GUIs. When you use a terminal you don't really care how many files are in a directory, unless they have names that are annoying to autocomplete.
The reason we bother with modules and directories is to help organize code and ease discovery. And whether we use ls or a GUI, we all start out the process of discovery with a directory listing.
The top-level directory, where these files are, is often where there's the most flexibility in terms of how to lay out things. All this clutter hides the project organization and makes it difficult to discover what's going on. It was one thing when it was just autotools, but the proliferation of tools in the last decade has brought it over a tipping point, IMHO.