Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web is missing an essential part of infrastructure: an open web index

arxiv.org

Post reply on HN
Print view View original

121–130 of 132 posts

Re: The Web is missing an essential part of infrastructure: an open web index

#121

Post by amelius »

amitamb wrote: post #72
AznHisoka wrote: post #8

Earlier quoted context omitted.

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

Apart from that Common Crawl respects robots.txt (which makes sense) so many sites you expect to see there are not indexed. Netflix, Facebook LinkedIn and many more. If common-crawl sees serious adoption those sites will modify their robots.txt but it's and chicken/egg problem.

There is a simple solution: if companies do not respect do-not-track then why should we respect robots.txt?

Re: The Web is missing an essential part of infrastructure: an open web index

#122

Post by papower »

throwamay1241 wrote: post #113
sneg55 wrote: post #111

Somebody could try to build own crawler and feed them with 260MM domain names dataset from https://domains-index.com

Is there more like this? Afaik SSL certificates are required to be committed to an open ledger but I can't find anywhere to obtain the ledger..

Maybe you're referring to Certificate Transparency Logs ? There's background info at http://www.certificate-transparency.org/what-is-ct

You could implement your own log monitor or use services like crt.sh or certstream to build a candidate list of domains that have registered SSL certs.

Re: The Web is missing an essential part of infrastructure: an open web index

#123

Post by mlinksva »

mbay wrote: post #110
mlinksva wrote: post #19

Could the Internet Archive, specifically https://web.archive.org/ be the basis of an Open Web Index as proposed by the author? I'm sure there are tons of obstacles to that path, but it also would be far ahead of any new initiative in at least two ways: it already has a huge index and ingestion pipeline, and it is a trusted organization.

yes - I worked on this a bit with Mark Graham, the director of the Wayback Machine

thx - can you say more?

Re: The Web is missing an essential part of infrastructure: an open web index

#124

Post by bogomipz »

asdfasgasdgasdg wrote: post #92
bogomipz wrote: post #86

Earlier quoted context omitted.

>"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data." Can you elaborate on what is the "simple correlated representation of the data"? It sounds like you understand this space pretty well might you have any links or literature on how modern crawling architectures and indexing work? Thanks.

Sorry, that's just a complicated way of saying that you can embed data in an inverted index that lets you guess how likely a document is to be found relevant on subsequent passes. Basically, you use various properties (embedded in the index) to filter down the list of documents you want to inspect in more detail, as you perform retrieval . There is some information in [1] about some types of filtering that can be don…

Thanks for the detailed explanation and the links, these are really helpful. Cheers.

Re: The Web is missing an essential part of infrastructure: an open web index

#126

Post by mdaniel »

known wrote: post #109
galaxyLogic wrote: post #63

How about indexing just the </h1? Is that the intention? We don't want too much information.

I prefer

Arguably no one invests in the title tag anymore because it's not user-visible in the way a heading tag is, or go further in the other direction and use the `` tags honored by Facebook and Twitter, since the page author has incentive to keep that content up-to-date

Re: The Web is missing an essential part of infrastructure: an open web index

#127

Post by icebraining »

russellbeattie wrote: post #68

The web needs to be forked into two distinct standards: One for dynamic content, and one for documents. The first would use basically everything in the HTML5/CSS/JS toolbox, and the second would be more akin to AMP, but for all docs. The benefits of this would be a standard for Wysiwyg editors (goodbye million rich text editor projects, Markdown and even Microsoft Word), and more semantic markup for both search engin…

> This isn't a technical challenge, it's an organizational one

No, it's an economical one. Who will use that web? You mention Twitter, yet are they not dependent on JS for analytics and ad-tracking? The few sites not dependent on such features are already usable on Lynx and Elinks, and the others simply won't use them.

For the advantages, you mention having a good WYSIWYG editor, but the reason you can't add bold or color to a Twitter post is obviously not because they are unable to add those functions, but because they don't want you to do that. Which raises the question: what happens when that editor lets you create something the site doesn't allow you to use?

(By the way, Wikipedia has had a visual editor since 2012, you just have to switch using the "pencil" button: https://en.wikipedia.org/wiki/Wikipedia:VisualEditor)

Re: The Web is missing an essential part of infrastructure: an open web index

#128

Post by icebraining »

lifeisstillgood wrote: post #20

I have argued that one regulatory outcome over Google could be the open release of their index - and even their database of "if you searched for X and clicked the top link then came back five seconds later we can infer the top link is not good for X" And yes I know that's pretty much all of Google. It's just that it's hard to get away from the idea that an index of web pages is anything other than the property of the…

You can't argue that such data is individually owned and also that it must be released publicly, because that would require consent from everyone whose data was used.

Re: The Web is missing an essential part of infrastructure: an open web index

#129

Post by mbay »

mlinksva wrote: post #123
mbay wrote: post #110

Earlier quoted context omitted.

yes - I worked on this a bit with Mark Graham, the director of the Wayback Machine

thx - can you say more?

Not too much really. Its a big interest of Mark's but its still early in the planning stages. I helped him with some preliminary research and gave this brief talk about our work: https://www.ischool.berkeley.edu/events/2018/facilitating-di...

Re: The Web is missing an essential part of infrastructure: an open web index

#130

Post by dredmorbius »

return1 wrote: post #67
dredmorbius wrote: post #61

Earlier quoted context omitted.

Suggestions as to how?

i dont know any relevant project sorry

Thanks, there's SubHub, and a few possible options.

An indexing standard seems a critical element.

Post reply on HN