Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web is missing an essential part of infrastructure: an open web index

arxiv.org

Post reply on HN
Print view View original

51–60 of 132 posts

Re: The Web is missing an essential part of infrastructure: an open web index

#51

Post by petra »

walterbell wrote: post #30
AndyPatterson wrote: post #28

Earlier quoted context omitted.

No idea where you got this statisti, but I guarantee no user would be happy to wait hours or (god forbid) days on a search result.

It's still sooner than "never", which is the current response time for answers that Google cannot provide. Central search indexes like Google are not going away. There are client-side metasearch interfaces that combine Google results with other sources. Those other sources can be much slower, including human responses. You would still have your synchronous sub-second response from centralized search, but there would…

>> Most of the world's knowledge is not public

Where is it found ?

Re: The Web is missing an essential part of infrastructure: an open web index

#52

Post by shereadsthenews »

petra wrote: post #51
walterbell wrote: post #30

Earlier quoted context omitted.

It's still sooner than "never", which is the current response time for answers that Google cannot provide. Central search indexes like Google are not going away. There are client-side metasearch interfaces that combine Google results with other sources. Those other sources can be much slower, including human responses. You would still have your synchronous sub-second response from centralized search, but there would…

>> Most of the world's knowledge is not public Where is it found ?

Not sure if this is what they meant but email is thought to be larger in aggregate than the web.

Re: The Web is missing an essential part of infrastructure: an open web index

#53

Post by Aeolun »

AznHisoka wrote: post #8
weinzierl wrote: post #6

Isn't this what Common Crawl[1] is. From their FAQ: > What is Common Crawl? > Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis. > What can you do with a copy of the web? > The possibilities are endless, but people have used the data to improve language translation sof…

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with your own more regular crawls.

Re: The Web is missing an essential part of infrastructure: an open web index

#54

Post by jpindar »

AndyPatterson wrote: post #28
walterbell wrote: post #22

Earlier quoted context omitted.

Google needs sub-second response to show ads. Some users may be happy to wait for hours or days to get high quality answers not available from commercial companies. Can still be faster than emailing a human friend or consultant or tasking an employee or department.

No idea where you got this statisti, but I guarantee no user would be happy to wait hours or (god forbid) days on a search result.

I can't be the only person who uses Google Alerts.

Re: The Web is missing an essential part of infrastructure: an open web index

#55

Post by bobajeff »

iamwil wrote: post #34

I didn't see mention of who would pay for this infrastructure. Is it considered a gov't funded or volunteer / donation thing? There doesn't seem to be a mention of how to alleviate a tragedy of the commons problem (unless I missed it). If common crawl is doing a fine job, who funds them?

The PDF near the end mentioned the EU as an example on who could fund it.

Re: The Web is missing an essential part of infrastructure: an open web index

#56

Post by snaky »

rabidrat wrote: post #36
zdragnar wrote: post #15

Earlier quoted context omitted.

If anything, that sounds like a solid argument to decentralize it. I don't want China's government, white supremacists, churches, soccer moms, Jihadis or grievance-of-the-month activists controlling how information is indexed; I would rather use multiple indexes that balance out controlling interests and biases.

Unfortunately if it's decentralized, then it becomes controlled by spamlords, SEO artists, advertisers, and anyone else who stands to gain from manipulating the index to their advantage. At least if it's centralized, the fights are out in the open and have a chance of converging on something reasonable (like e.g. wikipedia).

Decentralized doesn't mean flat. You can trust to some actors only (and to some they trust to).

Re: The Web is missing an essential part of infrastructure: an open web index

#57

Post by dredmorbius »

peterwwillis wrote: post #46
aheilbut wrote: post #43

I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.

Like a sitemap? https://en.m.wikipedia.org/wiki/Sitemaps

No, an actual search index, offerring word-to-URL mappings, along with metadata: creation and revision dates, authors, titles, file and/or MIME types, URLs within the text, other attributes, etc, from which a search interface could query.

Page-ranking would remain an issue, likely outside this scope.

I'd like to see some sort of cache-and-forward structure.

And you'd be relying on good-faith actors, which means heavily penalising bad actors.

Re: The Web is missing an essential part of infrastructure: an open web index

#58

Post by dredmorbius »

tokyodude wrote: post #10
michaelangerman wrote: post #7

I have been wanting this for years... If you look at the original Yahoo Page when Yahoo first started out it attempted to solve this problem. I believe this index could be regionally or language based... In the United States one could use Dewey Decimal https://en.wikipedia.org/wiki/Dewey_Decimal_Classification Library of Congress https://en.wikipedia.org/wiki/Library_of_Congress_Classifica...

It won't work without a central authority. See Soundcloud as an example. People tag their music with whatever they think will get them traffic. So, in order to do this you'll need a mass of volunteers which will lead to politics "XYZ should be classified as G! No it should be F!", "classifying ABC as DEF is racists/sexist/..." and other arguments. You'll also get people lobbying to have things removed (right to be fo…

No, you wouldn't need a central authority, though various subject indices and/or search interfaces (divorced from the crawl/index) would supply rank and/or reputation scores.

Re: The Web is missing an essential part of infrastructure: an open web index

#59

Post by lenzm »

aheilbut wrote: post #43

I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.

A significant problem with this is trust. You can't trust websites to reliably or accurately index their sites due to both incompetence and malice. I don't think there's any way around the malicious component. Formal or informal standards may take care of the competence factor with the feature being built into common publishing platforms.

XML sitemaps are a microcosm of putting the indexing onus on websites instead of the search engines - they are basically ignored by search engines because they have been abused and are not a useful signal. If pages aren't important enough to be linked to throughout your website then they aren't interpreted as being important enough to return to users. The optimistic case is that sitemaps/indices will send parallel signals to the search engines in which case they are redundant. The pessimistic case is that the sitemaps/indices will send signals orthogonal to the content provided to users in which case the website is either being deceitful or incompetent. In any case, the search engine will not want to use the sitemap/index as a signal as it either doesn't provide value or provides negative value.

Post reply on HN