Ask HN: Can we create a new internet where search engines are irrelevant? • Hacker News Board

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#261

Post by quelsolaar » Wed, Jun 26, 2019, 5:16 PM UTC

Yes, we need search engines, but they don't need to be monolithic. Imagine that indexing the text of your average web page takes up 10k. Then you get 100.000 pages per Gig. It means that you if you spend ~270USD on a consumer 10 tera drive you can index a billion webpages. Google no longer says how many pages they index, but its estimated to be with in one order of magnitude of that.

This means that in terms of hardware, you can build your own google, then you get to decide how it rates things and you don't have to worry about ads and SEO becomes much harder because there is no longer one target to SEO. Google obviously don't want you to do this (and in fairness google indexes a lot of stuff that isn't keywords form web pages), but it would be very possible to build an open source configurable search engine that anyone could install, run, and get good results out of.

(Example: The piratebay database, that arguably indexes the vast majority of avilable music / tv / film / software was / is small enough to be downloaded and cloned by users)

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#262

Post by ken » Wed, Jun 26, 2019, 5:18 PM UTC

Is this the same as asking if we can create a telephone system with no phone books, or a city with no maps? Where is our shared understanding of the system's state?

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#263

Post by gerbilly » Wed, Jun 26, 2019, 5:20 PM UTC

stakhanov wrote: post #259

iblaine wrote: post #228

Yes, it was called Yahoo and it did a good job of cataloging the internet when hundreds of sites were added per week: https://web.archive.org/web/19961227005023/http://www2.yahoo... I'm old enough to remember sorting sites by new to see what new URLs were being created, and getting to that bottom of that list within a few minutes. Google and search was a natural response to solving that problem as the number of sites…

You don't have to go all the way back into Yahoo-era when it comes to manually curated directories: DMOZ was actively maintained until quite recently, but ultimately given up for what seems like good reasons.

Earlier than that there was a list of ftp sites giving a summary of what was available on each.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#264

Post by wjmao88 » Wed, Jun 26, 2019, 5:21 PM UTC

A similar, simplified form of this question is: How would you navigate a large code base without using an IDE? I think it would have a high requirement on the structure of the files and the architecture of the code, neither of which can be achieved with what we have on the internet today.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#265

Post by alangibson » Wed, Jun 26, 2019, 5:21 PM UTC

iblaine wrote: post #228

Yes, it was called Yahoo and it did a good job of cataloging the internet when hundreds of sites were added per week: https://web.archive.org/web/19961227005023/http://www2.yahoo... I'm old enough to remember sorting sites by new to see what new URLs were being created, and getting to that bottom of that list within a few minutes. Google and search was a natural response to solving that problem as the number of sites…

I wonder if you could build a Yahoo/Google hybrid where you start with many trusted catalogs run by special interest groups then index only those sites for search. Doesn't fully solve the centralization problem, but interesting none the less.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#266

Post by [deleted] » Wed, Jun 26, 2019, 5:23 PM UTC

iblaine wrote: post #228

Yes, it was called Yahoo and it did a good job of cataloging the internet when hundreds of sites were added per week: https://web.archive.org/web/19961227005023/http://www2.yahoo... I'm old enough to remember sorting sites by new to see what new URLs were being created, and getting to that bottom of that list within a few minutes. Google and search was a natural response to solving that problem as the number of sites…

[deleted]

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#267

Post by munchausen42 » Wed, Jun 26, 2019, 5:24 PM UTC

To get rid of search engines like Google and Bing we don't need to build a new internet - we just need to build new search engines.

E.g., how about an open source spider/crawler that anyone can run on their own machine continuously contributing towards a distributed index that can be queried in a p2p fashion. (Kind of like SETI@home but for stealing back the internet).

Just think about all the great things that researchers and data scientists could do if they had access to every single public Facebook/Twitter/Instagram post.

Okayokay ... also think about what Google and FB could do if they could access any data visible to anyone (but let's just ignore that for a moment ;)

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#268

Post by asdff » Wed, Jun 26, 2019, 5:25 PM UTC

adrianmonk wrote: post #257

I think it would be helpful to remember to distinguish two separate search engine concepts here: indexing and ranking. Indexing isn't the source of problems. You can index in an objective manner. A new architecture for the web doesn't need to eliminate indexing. Ranking is where it gets controversial. When you rank, you pick winners and losers. Hopefully based on some useful metric, but the devil is in the details on…

Not to mention all the people who will carefully study whatever new system, looking for their angle to game the ranking.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#269

Post by fooker » Wed, Jun 26, 2019, 5:25 PM UTC

I'll be pessimistic here and say no, that is an impossible pipe dream. For any such system design you can come up with, a centralized big brother controller system will be more more efficient and have better user experience.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#270

Post by IanSanders » Wed, Jun 26, 2019, 5:28 PM UTC

TheOtherHobbes wrote: post #219

IanSanders wrote: post #189

Earlier quoted context omitted.

I think heavy reliance on human language (and its ambiguity) is one of the main problems. Maybe personal whitelist/blacklist for domains and authors could improve things. Sort of "Web of trust" but done properly. Not completely without search engines, but for example, if every website was responsible for maintaining it's own index, we could effectively run our own search engines after initialising "base" trusted webs…

This would be ludicrously easy to game. Crowdsourcing would also be ludicrously easy to game. The problem isn't solvable without a good AI content scraper. The scraper/indexer either has to be centralised - an international resource run independently of countries, corporations, and paid interest groups - or it has be an impossible-to-game distributed resource. The former is hugely challenging politically, because the…

User's search engine doesn't have to trust suggestions verbatim, it can always run its own heuristic on top of returned results. And the user could reduce the weight of especially uncooperative domains or blacklist them altogether.