>
[N]ot everyone deserves an equal say - that invalidates the whole idea of differences in skill/knowledge among domains across people.What makes me mad is that this isn't even that controversial; people just go straight to "completely offended" about so many things nowadays without accepting that they've leveled-up as much as they have, that discussing things competently must be earned, and that the right to do so should come with the ability.
"Representative democracies" sounds like scalability-speak :P - where I define "scalability" to mean "dropping tons of rounding errors on the floor". (Cuz that's what invariably happens.)
Network "smallness" wasn't really a concrete enough explanation for "works better" to me; I knew there was something extra to be said besides "human networking doesn't scale". This is of course true, but too generalized IMO.
Niche interest provides a slightly more concrete explanation: you're right, only people super-interested in relevant fields (or recommended by word of mouth, perhaps) used to find HN.
And yes, now we have spammers of all kinds. HN is full of the weird ASCII soup ones, I've noticed tons of them of late sadly. But the noise here is greater, and sadly lots of interesting content gets firmly buried the moment it's posted while fluff rises to the top. I wonder how much of the fluff is engineered.
As for a solution to this, I think whole-system analysis could be useful - scraping e.v.e.r.y.t.h.i.n.g on a site, all the content and even images, and then being able to cross-reference everything, shove it through arbitrary models, etc. Having everything in once place is I think the key part - this is after all how our brains work, and since this is about human networking I wonder if this isn't at least a good approach to start with.
Nobody seems to be doing this at scale for eg reddit or HN, even though the data is available (reddit has dumps up on bigquery and there are also torrents; and HN's firebase API is officially explicitly non-ratelimited, confirmed via email). Beyond that, looking at for example all of tumblr is something I've wanted to do for a very long time.
Ultimately it's about being able to lose the right information in the summarization process. Summarization is always lossy; but if you lose low-value info then the essence can be reconstructed. (If I summarize this message by dropping everything except for "if" "and" "a" "as" "for" and "I" (etc), the result would be non-reconstructable, for example.)
"Big data" is currently at the pattern-matching stage. The stage I've just described - summarizing consistently - will be the one where people will be falling over each other with new uses etc etc. I think everyone's anthropomorphizing the stage we're at, thinking we're already doing summarization, because it's how we intuitively interpret the idea of pattern-matching (when we aren't actively thinking about the mechanics of it). But we aren't yet.
So for now, we need to do this ourselves, I think. I've been thinking about tagging for many years; del.icio.us absolutely had the right idea with that, it's so sad it's now a broken personal-use-only archive (it got bought by pinboard and is now effectively non-scrapeable D:).
I'm hoping to put a forum of my own together at some point. Considering the speed I do just about anything and the number of ideas I know I want to fold in I know the earliest I'm likely to have anything interesting is probably this time next year (ha). It uses tagging as one of its foundations along with a few other tidbits. (Of course they is sekret, I want the forum to be a surprise :D)
I also have some ideas about web scraping I want to play with at some point. Well, not really web scraping, but more a "bigger picture" type thing that makes scraping interesting/useful. _That's_ a 10-year project, one I've been thinking about since 2003. (I still haven't started it, as I'm still figuring it out. It's not rocket science, it's just getting the intersection of a thousand tiny details right.)
Problem #1 is articulating everything - I have no idea how to describe all my ideas because the only way I know how to write is to describe things with a beginning middle and end. But there's no middle or end because this hasn't happened yet (lol). So I go to write about this and I get completely stuck on the "beginning" part, describing what I'm about to describe in an infinite loop. Hilarious and depressing all at once.
I've recently realized mind-mapping could help there, now all I need to do is get (or build) a mind-mapping system that'll work on my ancient laptop... ("what's a GPU?")