When I picture it in my head I think of the early web as more of a library. Over time it has transitioned into a shopping mall. If I continue with this thought exercise, a lot of the big indoor shopping malls around me have been knocked down and replaced with standalone outdoor stores (walled gardens?). I'm not sure where things are going next.
I think there have been two main changes that have hurt the web: 1) The shift from spammy shit content as something to squash to something to allow and even promote over better content , provided it follows certain (Google's) rules. This shift (in terms of Google's behavior) happened ~2008-2010 and we haven't seen a period of spammy crap content getting heavily downranked since then, like we used to when they were st…
One development that would be good for small web sites to look into is schema.org linked-data formats. Those might simply be too effort-intensive for the spammers to adopt (at a high level of detail) and perhaps too much of a commitment to quality and transparency (they would have to actively forge the info, which would leave them open to bans given the lack of plausible deniability), so they might become a viable signal of quality and lead to higher visibility in SERP.
(Similar for things like proper separation of style from content, that have always been advocated for in the web-standards community but are not really commercially viable.)
I'm not quite sure if others have experimented with this stuff already, but it seems worth trying.