Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web is missing an essential part of infrastructure: an open web index

arxiv.org

Post reply on HN
Print view View original

81–90 of 132 posts

Re: The Web is missing an essential part of infrastructure: an open web index

#81

Post by 3xblah »

The PDF is a little short on details. It sounds like webamsters would all have to cooperate with allowing crawls from an "OWI" bot.

One of the challenges of creating a "web index" is first creating indexes of each website. "Crawling" to discover every page of a website, as well as all links to external sites, is labour-intensive and relatively inefficient. Part of that is because there is no 100% reliable way to know, before we begin accessing a website, each and every URL for each and every page of the site. There are inconsistent efforts such "site index" pages or the "sitemap" protocol (introduced by Google), but we cannot rely on all websites to create a comprehensive list of pages and to share it.

However, I believe there is a way to generate such a list from something that almost all websites do create: logs.

When Google crawls a website, it is often or maybe even always the case that the site generates logs of every HTTP request that googlebot makes.

If a website were to share publicly, in some standardised format, the portion of their log where googlebot has most recently crawled the site, we might see a URL for each and every page of the site that Google has requested.

Automating this procedure of sharing listings of those googlebot HTTP requests, the public could generate a "site index" directly from the source, via the information on googlebot requests in the logs.

Allowing crawls from a "new" bot would not be necessary.

Webmasters know what URLs they offer to Google. Google knows as well. The public, however, does not.

It is a public web. Absent mistakes by webmasters, any pages that Google is allowed to crawl are intended to be public.

Why should the public not have access to a list of all the pages of websites that Google crawls?

I don't know, but there must be reasons I have failed to consider.

What are the reasons the public not know what pages are publicly available via the web, except as made visible (or invisible) through a middleman like Google?

There are none.

Being able to see logs of all the googlebot requests would be one way to see what Google has in their index without actually accessing Google.

Re: The Web is missing an essential part of infrastructure: an open web index

#82

Post by yomly »

3xblah wrote: post #81

The PDF is a little short on details. It sounds like webamsters would all have to cooperate with allowing crawls from an "OWI" bot. One of the challenges of creating a "web index" is first creating indexes of each website. "Crawling" to discover every page of a website, as well as all links to external sites, is labour-intensive and relatively inefficient. Part of that is because there is no 100% reliable way to know…

Isn't the act of sharing these logs vulnerable to a similar problem to site maps?

Not everyone will do it and those that do may not do it to 100% completeness: people may not keep their http logs in good order, for example.

Re: The Web is missing an essential part of infrastructure: an open web index

#83

Post by bogomipz »

Aeolun wrote: post #53
AznHisoka wrote: post #8

Earlier quoted context omitted.

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with your own more regular crawls.

I'm curious how they track and rank a site's "change velocity" without crawling all of the internet all of the time. It almost seems like a catch 22 no? Might you have any insight into how this works? Any suggested reading or links?

Re: The Web is missing an essential part of infrastructure: an open web index

#84

Post by shereadsthenews »

Ironically it is EU regulations that make this idea totally impossible. One does not simply index documents, at least not for Europeans. You have to expurgate your index for the "right to be forgotten" people. You have to remove all the Nazi stuff because of Germans. This idea by a German is not possible because Europe.

Re: The Web is missing an essential part of infrastructure: an open web index

#85

Post by bogomipz »

oldjokes wrote: post #77
lenzm wrote: post #59

Earlier quoted context omitted.

A significant problem with this is trust. You can't trust websites to reliably or accurately index their sites due to both incompetence and malice. I don't think there's any way around the malicious component. Formal or informal standards may take care of the competence factor with the feature being built into common publishing platforms. XML sitemaps are a microcosm of putting the indexing onus on websites instead o…

Honestly I'd much rather have a bunch of dice rolls on incompetence than the current centralized, single point of control over the entire index. Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using. It's difficult to imagine a worse solution for the web than this current model.

"Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using."

Wow, interesting, this is the first I've heard of this. Might you have some link or citations about this? Thanks.

Re: The Web is missing an essential part of infrastructure: an open web index

#86

Post by bogomipz »

asdfasgasdgasdg wrote: post #70
aheilbut wrote: post #43

I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.

I think there are a few factors that would make this idea unworkable. There are two categories of issues, technical and economic, that prevent this from working. I'll go into more detail about the technical issues. The top-level problem is fan-out. If you want to fan the query to the top million domains (far too few to match Google's retrieval depth, but enough to demonstrate the issue), you're going to need to imple…

>"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data."

Can you elaborate on what is the "simple correlated representation of the data"? It sounds like you understand this space pretty well might you have any links or literature on how modern crawling architectures and indexing work? Thanks.

Re: The Web is missing an essential part of infrastructure: an open web index

#88

Post by ddevault »

burtonator wrote: post #80
AznHisoka wrote: post #8

Earlier quoted context omitted.

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

This is literally why I created my company: http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline ex…

This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.

Re: The Web is missing an essential part of infrastructure: an open web index

#89

Post by 3xblah »

yomly wrote: post #82
3xblah wrote: post #81

The PDF is a little short on details. It sounds like webamsters would all have to cooperate with allowing crawls from an "OWI" bot. One of the challenges of creating a "web index" is first creating indexes of each website. "Crawling" to discover every page of a website, as well as all links to external sites, is labour-intensive and relatively inefficient. Part of that is because there is no 100% reliable way to know…

Isn't the act of sharing these logs vulnerable to a similar problem to site maps? Not everyone will do it and those that do may not do it to 100% completeness: people may not keep their http logs in good order, for example.

"Not everyone will do it..."

Not everyone will provide CCBot with the same access that they provide to Googlebot. The question is how many will?

It is sort of a catch-all issue with anything on the web: "Not everyone will do it." I am not sure that anyone aims for 100% participation where the web is concerned.

There is always an uncertain amount of variation involved with particpation in anything across the entire www.

Re: The Web is missing an essential part of infrastructure: an open web index

#90

Post by losteric »

burtonator wrote: post #80
AznHisoka wrote: post #8

Earlier quoted context omitted.

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

This is literally why I created my company: http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline ex…

Cheaper than the competition? Maybe. Nothing that requires contact to get a price is "affordable" (if you have to ask, you can't afford it...)
Post reply on HN