You could make a browser plugin that effectively turned everyone into a spider that sent new chunks of the index to some decentralized blockchain-esque storage system for all to query with its own blockchain-esque micro payments
Ask HN: Can we create a new internet where search engines are irrelevant?
251–260 of 395 posts
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#252Indexing information is a political problem as much as a technical one. Ultimately there will always be people who will put more effort into getting their information known than others. These people would game whatever technical solution exists.
Thats true. What if you could artificially limit the amount of effort someone can put in to getting content out there? Or even make it known to the consumer how/why the content is ranked highly?
Most closed platforms do a subset of what you mention: I can only put so many posts on my Facebook before they stop making it to all my friends. If I pay more for higher ranking it’s labeled an advertisement.
However, creating rules transitions the contention point to who makes the rules. If you think that my algorithm will rank my sources better than your sources, you may be less interested in my algorithm regardless of its technical merits.
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#253Yes, it was called Yahoo and it did a good job of cataloging the internet when hundreds of sites were added per week: https://web.archive.org/web/19961227005023/http://www2.yahoo... I'm old enough to remember sorting sites by new to see what new URLs were being created, and getting to that bottom of that list within a few minutes. Google and search was a natural response to solving that problem as the number of sites…
I used Yahoo back in those days, and it literally proved the point that hand-cataloging the internet wasn't tractable, at least not the way Yahoo tried to do it. There was just too much volume. It was wonderful to have things so carefully organized, but it took months for them to add sites. Their backlog was enormous. Their failure to keep up is basically what pushed people to an automated approach, i.e. the search e…
I found myself briefly wondering if it were possible to have a decentralized open source repository of curated sites that anyone could fork, add to, or modify. Then I remembered dmoz, which wasn't really decentralized -- and realized that "awesome lists" on GitHub may be a critical step in the direction I had envisioned.
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#254Indexing information is a political problem as much as a technical one. Ultimately there will always be people who will put more effort into getting their information known than others. These people would game whatever technical solution exists.
Thats true. What if you could artificially limit the amount of effort someone can put in to getting content out there? Or even make it known to the consumer how/why the content is ranked highly?
Or allow users to specify precisely how to weight the results? (Popular with linkers, best matching results first, etc.)
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#255That was what the early internet was like (I was there). People built indexes by hand, lists of pages on certain topics. There was the Gopher protocol that was supposed to help with finding things. But this was all top-down stuff, the first indexing/crawling search engines were bottom-up and it worked so much better. And for a while we had an ecosystem of different search engines until Google came along, was genuinel…
In the very early days, you didn't need a search engine because there weren't that many web sites and you knew most of the main ones anyway (or later on had them in your own hotlists in Mosaic). Nowadays you need a search because there is so much content. The problem is that the amount of content and the size of the potential user base are so large that is is impossible to offer search as a free service, i.e. it has…
Exactly my thought. But it definitely wouldn't get mass adoption which is good because mass-market content websites are questionable in terms of user experience (they also need to cover content creating costs by popups/ads/pushes). One thing, though, ad based search engines lift ad based websites because they can sell ad on a second end.
Maybe we'll see advent of specialised paid search engines SaaSs with authentic and independent content authors like professional blogs.
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#256Earlier quoted context omitted.
The problem is that current search engines are indexing what is essentially a stack of random books thrown together by anonymous library goers. Before being able to guide readers to books, librarians have to the following non-trivial tasks over the entire collection: - identify the book's theme - measure the quality of the information - determine authenticity / malicious content - remember the position of the book in…
>I welcome any attempts to build a more organized internet. I don't think the communal book pile approach is scaling very well. Let me know if I misunderstand your comment but to me, this has already been tried. Yahoo's founders originally tried to "organize" the internet like a good librarian. Yahoo in 1994 was originally called, "Jerry and David's Guide to the World Wide Web" [0] with hierarchical directories to cu…
Consider the sheer size of the internet now. Even if you could categorize and file that many websites accurately, how do you display that to the user in a way that's usable? It will probably look a lot like a search engine, no matter which way you frame it.
The underlying goal: "Get a user the information they want when they don't know where it lives" isn't really going to be helped by a non-searchable directory of millions of sites.
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#257I think it would be helpful to remember to distinguish two separate search engine concepts here: indexing and ranking.
Indexing isn't the source of problems. You can index in an objective manner. A new architecture for the web doesn't need to eliminate indexing.
Ranking is where it gets controversial. When you rank, you pick winners and losers. Hopefully based on some useful metric, but the devil is in the details on that.
The thing is, I don't think you can eliminate ranking. Whatever kind of site(s) you're seeking, you are starting with some information that identifies the set of sites that might be what you're looking for. That set might contain 10,000 sites, so you need a way to push the "best" ones to the top of the list.
Even if you go with a different model than keywords, you still need ranking. Suppose you create a browsable hierarchy of categories instead. Within each category, there are still going to be multiple sites.
So it seems to me the key issue isn't ranking and indexing, it's who controls the ranking and how it's defined. Any improved system is going to need an answer for how to do it.
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#258Earlier quoted context omitted.
I think heavy reliance on human language (and its ambiguity) is one of the main problems. Maybe personal whitelist/blacklist for domains and authors could improve things. Sort of "Web of trust" but done properly. Not completely without search engines, but for example, if every website was responsible for maintaining it's own index, we could effectively run our own search engines after initialising "base" trusted webs…
This would be ludicrously easy to game. Crowdsourcing would also be ludicrously easy to game. The problem isn't solvable without a good AI content scraper. The scraper/indexer either has to be centralised - an international resource run independently of countries, corporations, and paid interest groups - or it has be an impossible-to-game distributed resource. The former is hugely challenging politically, because the…
Just want to point out that you're on a site that successfully uses crowd sourcing combined with moderation to curate a list of websites, news, and articles that people find interesting and valuable. Why not a new internet built around communities like this where the users actively participate in finding, ranking, and moderating the content they consume? It's not a stretch to add a decent search index and categories to a news aggregator, most do it already. If these tools could be built into the structure of the web we'd be half way there.
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#259Yes, it was called Yahoo and it did a good job of cataloging the internet when hundreds of sites were added per week: https://web.archive.org/web/19961227005023/http://www2.yahoo... I'm old enough to remember sorting sites by new to see what new URLs were being created, and getting to that bottom of that list within a few minutes. Google and search was a natural response to solving that problem as the number of sites…
You don't have to go all the way back into Yahoo-era when it comes to manually curated directories: DMOZ was actively maintained until quite recently, but ultimately given up for what seems like good reasons.
Re: Ask HN: Can we create a new internet where search engines are irrelevant?
#260Earlier quoted context omitted.
The problem with git is countering nefarious forces. The blockchain is better in that regard because the consensus algorithm can be used to verify that the listings are legitimate.
content change signed by creators private key, otherwise merge is rejected? or, wiki approach...
Just signing with a private key isn't a guarantor of anything other than that if you trust that the person with the key is who they say they are, then the actual content is from them. But that would require a massively large web of trust in itself: that all the private keys would be trusted. And if you only let in private keys that you explicitly trusted, then it's very likely you could end up with an echo chamber