Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web is missing an essential part of infrastructure: an open web index

arxiv.org

Post reply on HN
Print view View original

91–100 of 132 posts

Re: The Web is missing an essential part of infrastructure: an open web index

#91

Post by ape4 »

AznHisoka wrote: post #8
weinzierl wrote: post #6

Isn't this what Common Crawl[1] is. From their FAQ: > What is Common Crawl? > Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis. > What can you do with a copy of the web? > The possibilities are endless, but people have used the data to improve language translation sof…

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

Yup, article says "A search engine needs to keep its index current, meaning it needs to update at least a part of it every minute. This is an important requirement that is not being met by any of the current projects (like Common Crawl) aiming at indexing snapshots of (parts of) the Web."

Re: The Web is missing an essential part of infrastructure: an open web index

#92

Post by asdfasgasdgasdg »

bogomipz wrote: post #86
asdfasgasdgasdg wrote: post #70

Earlier quoted context omitted.

I think there are a few factors that would make this idea unworkable. There are two categories of issues, technical and economic, that prevent this from working. I'll go into more detail about the technical issues. The top-level problem is fan-out. If you want to fan the query to the top million domains (far too few to match Google's retrieval depth, but enough to demonstrate the issue), you're going to need to imple…

>"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data." Can you elaborate on what is the "simple correlated representation of the data"? It sounds like you understand this space pretty well might you have any links or literature on how modern crawling architectures and indexing work? Thanks.

Sorry, that's just a complicated way of saying that you can embed data in an inverted index that lets you guess how likely a document is to be found relevant on subsequent passes. Basically, you use various properties (embedded in the index) to filter down the list of documents you want to inspect in more detail, as you perform retrieval. There is some information in [1] about some types of filtering that can be done (e.g. their discussion on tiered retrieval and the notion of a global quality score being used to discard candidates). Lucene calls these bits of index-embedded data "token attributes," [2] but how exactly they are used depends on the scorer implementation. To learn more about how the industry approaches these issues, unfortunately you have to join one of the companies that's on the leading edge of this type of research, since they are loathe to disclose too much.

[1]: https://nlp.stanford.edu/IR-book/pdf/07system.pdf

[2]: https://pdfs.semanticscholar.org/2795/d9d165607b5ad6d8b97183...

Re: The Web is missing an essential part of infrastructure: an open web index

#93

Post by walterbell »

burtonator wrote: post #80
AznHisoka wrote: post #8

Earlier quoted context omitted.

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

This is literally why I created my company: http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline ex…

Have you considered making a subset of your data open, cross-referenced from the paid data set? If other providers followed this approach, the open data set could grow and become more useful to all of the paid data providers, if only for lead generation and tool interoperability.

Re: The Web is missing an essential part of infrastructure: an open web index

#94

Post by jdsully »

bogomipz wrote: post #83
Aeolun wrote: post #53

Earlier quoted context omitted.

Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with your own more regular crawls.

I'm curious how they track and rank a site's "change velocity" without crawling all of the internet all of the time. It almost seems like a catch 22 no? Might you have any insight into how this works? Any suggested reading or links?

A site you just learned about probably isn't very important, so you can just watch its month to month change. You don't care if they change that much because they aren't important yet.

A site that gets lots of links quickly (and is therefore important) will likely garner them from sites you are already frequently visiting.

Re: The Web is missing an essential part of infrastructure: an open web index

#95

Post by la6470 »

ddevault wrote: post #88
burtonator wrote: post #80

Earlier quoted context omitted.

This is literally why I created my company: http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline ex…

This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.

You are a fool ... it takes money to run servers pay for electricity and cooling and salaries for the people who work for it. Don’t talk like a child.

Re: The Web is missing an essential part of infrastructure: an open web index

#96

Post by lawrenceyan »

burtonator wrote: post #80
AznHisoka wrote: post #8

Earlier quoted context omitted.

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

This is literally why I created my company: http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline ex…

How exactly is it open if you have a paywall blocking people from accessing it though?

Re: The Web is missing an essential part of infrastructure: an open web index

#97

Post by peterburkimsher »

michaelangerman wrote: post #7

I have been wanting this for years... If you look at the original Yahoo Page when Yahoo first started out it attempted to solve this problem. I believe this index could be regionally or language based... In the United States one could use Dewey Decimal https://en.wikipedia.org/wiki/Dewey_Decimal_Classification Library of Congress https://en.wikipedia.org/wiki/Library_of_Congress_Classifica...

I agree that regions & languages are one way to classify data, but there are other more meaningful sub-culture categories.

I tried to simplify all data into ~30 categories. My own interests fit into 16, so I drew a visual representation of them. https://github.com/peterburk/sortlikes

Next, I need to figure out the sub-categories. Genres for music, countries for travel, etc.

What interests me most is the cross-cultural connections. For example, Taiwanese punk rock (Fire Ex), or Mongolian folk metal (Nine Treasures, Hanggai). I like that music because it's the same sub-category I'm interested in (Music/Rock).

It's also possible to model the flow of finance around the world through this categorisation. Some of the categories are innately human and don't seem to exist in animals (music, cooking).

Email me if you'd like to chat more about how to categorise culture - I think it's important and I've got lots of ideas about it, but I haven't yet met any other people with this same passion.

Re: The Web is missing an essential part of infrastructure: an open web index

#98

Post by p49k »

burtonator wrote: post #80
AznHisoka wrote: post #8

Earlier quoted context omitted.

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

This is literally why I created my company: http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline ex…

If your prices are so much more reasonable than competition, why are they not published publicly on your site? “Contact us and we’ll tell you the price” is shady for a service that claims to be “very very very affordable.”

Re: The Web is missing an essential part of infrastructure: an open web index

#99

Post by peterburkimsher »

diNgUrAndI wrote: post #73

As a user, if some other search engine can serve results that are better than Google, I'd be happy to use it. I've tried duckduckgo, the results are disappointing and often mis-intepreted what I intended to search. So I kept coming back to Google. Will Google be willing to open its indexes? Probably not at their best interest, because it will help its competitors?

I had an idea about a new indexing algorithm that would only need static file hosting (e.g. Github) for searching.

https://news.ycombinator.com/item?id=17548623

If you like, I can try implementing that with my next data analysis project. Right now I'm studying the MySpace Dragon Hoard, and I'll soon write a blog post with maps of music genres around the world.

Re: The Web is missing an essential part of infrastructure: an open web index

#100

Post by sytelus »

ddevault wrote: post #88
burtonator wrote: post #80

Earlier quoted context omitted.

This is literally why I created my company: http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline ex…

This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.

You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope these indices are available on AWS, Azure etc where people can just use it with cloud compute and pay per use.
Post reply on HN