I know what you are saying and I agree and am grateful for it's existance but keep in mind that HN is a marketing tool of a big VC fund. It has money to pay a moderator and doesn't need to directly make money from it by putting ads or tracking scripts up like other sites.
Yes, that's the sweet spot that HN ended up in, kind of by historical accident, and it seems to be a local optimum.
If we had to make money from HN in any of the standard squeezy ways, it would get worse, which would degrade if not ruin the community. It would also be a miserable slog and who would want to work on that? so the quality would go down that way too. There are a lot of gradients along which the quality would go down.
Because we don't have to do that, we can focus on just trying to keep the community happy, which boils down to trying to keep HN as good as possible. It's incredibly satisfying to just have to make something good. (Not that it is good...but that's a separate question, and this is the internet and we get graded on a curve.)
There are a bunch of old comments here about how we/I think about HN vis-à-vis YC's business interests, if anyone is curious - https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so....