Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wouldn't it be fun to build your own Google?

radar.oreilly.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Wouldn't it be fun to build your own Google?

#31

Post by Smerity »

jtoy wrote: post #27
Smerity wrote: post #10

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it…

anyway we can get an open sourced version of that optimized job for us to play with?

There's are two levels of optimizations that come into play: AWS setup and the choice of primary language. I'm always happy to speak about both as we love seeing experiments run over the data!

AWS optimizations: For that level of cost efficiency, you really need to use spot instances. A cluster of 100 m1.xlarge machines (1.5TB of RAM, 400 cores, and 168TB of magnetic disk storage) will only cost you $3 per hour using spot instances, rather than $30 on-demand. You should pay on-demand prices for the Hadoop master however.

You'll also want to roll your own Hadoop cluster as opposed to using Elastic MapReduce (EMR). EMR is amazing but the cost overhead when using it on spot instances is ~100%.

For the code itself, this is a situation where you'll want to stick with the programming language that has both the best performance and best ecosystem. I'm personally not a big fan of Java, but it really does win out here -- it's close to C or C++ for performance and has the advantage of the Hadoop ecosystem behind it. Other languages are certainly usable, but even if LanguageX only ran 4x slower than Java, the resulting job would be 4x more expensive due to paying by the hour.

Other than that, it's really just a standard MapReduce job using Hadoop. You can see an three examples for the three different data formats we use at: https://github.com/commoncrawl/cc-warc-examples/

Re: Wouldn't it be fun to build your own Google?

#32

Post by elyase »

Smerity wrote: post #31
jtoy wrote: post #27

Earlier quoted context omitted.

anyway we can get an open sourced version of that optimized job for us to play with?

There's are two levels of optimizations that come into play: AWS setup and the choice of primary language. I'm always happy to speak about both as we love seeing experiments run over the data! AWS optimizations: For that level of cost efficiency, you really need to use spot instances. A cluster of 100 m1.xlarge machines (1.5TB of RAM, 400 cores, and 168TB of magnetic disk storage) will only cost you $3 per hour using…

Thanks for your comments. I see you get 400 cores for 3USD/h . Where you optimizing for processing power per dollar? Do you know if the m1.xlarge instances are the best for this use case?

Re: Wouldn't it be fun to build your own Google?

#33

Post by Smerity »

elyase wrote: post #32
Smerity wrote: post #31

Earlier quoted context omitted.

There's are two levels of optimizations that come into play: AWS setup and the choice of primary language. I'm always happy to speak about both as we love seeing experiments run over the data! AWS optimizations: For that level of cost efficiency, you really need to use spot instances. A cluster of 100 m1.xlarge machines (1.5TB of RAM, 400 cores, and 168TB of magnetic disk storage) will only cost you $3 per hour using…

Thanks for your comments. I see you get 400 cores for 3USD/h . Where you optimizing for processing power per dollar? Do you know if the m1.xlarge instances are the best for this use case?

It really depends on the use case. I mentioned m1.xlarge as they provide a good general cluster setup. The mix of CPU, RAM, and disk space should work well for most experiments one might want to perform. m1.xlarge instances also have 1Gbps network interfaces when others in the same price range have 500Mbps -- a vestige of the older generation of machines. Finally, they excel at disk space. If you're utilizing HDFS heavily, newer instances are usually SSD (good) but have 10 to 20 times less disk storage (bad).

For the same dollar amount, you can trade for other specs though.

If you're more interested in CPU / RAM / SSD for example, paying $3USD/h for r3.xlarge gets you 3TB of RAM, about 1.5 times more computing power (same number of cores but more compute units), but far less disk space -- 8TB of SSD.

In the end, it really depends on the task at hand, but your dollar does go quite far regardless!

Re: Wouldn't it be fun to build your own Google?

#34

Post by jobposter1234 »

curiously wrote: post #20
Smerity wrote: post #10

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it…

what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their human uploaded content and profiles, and majority of the data from the web being useless without a context and purpose of the searcher? I'm just curious as to how commoncrawl compares with kimonolabs and import.io as they seem to have the same goal of creating…

Regarding copyright issues, I believe you can still use copyrighted data as long as it's transformed. E.g., building language models, or doing a search engine like Google. In fact, I can think of more computational uses for copyrighted data, while on the "banned" side, I can only think of... SEO.

Regarding point two: "monopoly businesses who viciously protect their human uploaded content". I spend a lot of time scraping these monopoly businesses, and it seems to me they do a decent job of letting their users decide what data is exposed. Facebook, linkedin, and google all are decent about letting me scrape their public info. That's all I have a right to -- private info should stay private, at the behest of the owner (the User in UCG).

You are correct regarding the third point, but I don't see that as a problem. This isn't a solution in search of a problem -- it's a problem without a solution at the moment.

Here's a toy example of something I'd like to do: calculate the positive / negative sentiment of commenters at particular baseball fan sites, so I can hide the content I don't like, and show that which I do. Having a common crawl of the site would be immensely useful (and is indeed a prereq) for this. I wouldn't need to republish it, just compute on it.

Re: Wouldn't it be fun to build your own Google?

#35

Post by jobposter1234 »

threeseed wrote: post #29
discardorama wrote: post #16

Google's power comes not from the crawling, but from the retrieval and ranking. They use many more signals than the hyperlinks and anchor text (which is all you'd have if you crawled yourself). Indexing crawled content would have been OK in the year 2000; but today, the users demand more. Relevance is the top priority, and no one does it better than El Goog.

Sorry but Google's ranking algorithm for me is far from brilliant. To give you an example, search for "webhcat primary key" (without quotes) and note how the top three search results do not actually contain the term webhcat. Google constantly does this. It randomly ignores search terms unless you explicitly quote them. I believe that there is still a market for a technical/advanced search engine.

Isn't google doing that because it detected the semantic information was on the page, even if the exact term wasn't? Is your issue with the fact that they're doing more than just a keyword retrieval, or is your issue with the fact that they're doing it poorly?

Re: Wouldn't it be fun to build your own Google?

#36

Post by smoyer »

A couple thoughts:

1) I like the idea of human curation, but in combination with some sort of automated crawler (or other tool) that helps in the browser.

2) Why can't we also distribute the act of crawling, the maintenance of the index and the map-reduce (or other algorithm) that produces the data.

I've been thinking about architectures that would allow (in essence) a P2P search system. Would anyone be interested in talking about architectures to make this work? There are millions of computers on the web at any given time ... if it's built into the browser (or plugs in), you could have human input at the same time.

Re: Wouldn't it be fun to build your own Google?

#37

Post by threeseed »

jobposter1234 wrote: post #35
threeseed wrote: post #29

Earlier quoted context omitted.

Sorry but Google's ranking algorithm for me is far from brilliant. To give you an example, search for "webhcat primary key" (without quotes) and note how the top three search results do not actually contain the term webhcat. Google constantly does this. It randomly ignores search terms unless you explicitly quote them. I believe that there is still a market for a technical/advanced search engine.

Isn't google doing that because it detected the semantic information was on the page, even if the exact term wasn't? Is your issue with the fact that they're doing more than just a keyword retrieval, or is your issue with the fact that they're doing it poorly?

Isn't my issue obvious ? I wanted search results that contained the search terms. Otherwise I wouldn't have entered them in the first place.

I understand Google is trying to be clever here and appealing to novices who don't really understand what they want.

But my point is that for those of us that do it is an incredibly annoying "feature". Feature is in quotes because in the specific case above they didn't find semantic equivalents. They just dropped the "webhcat" term entirely.

Re: Wouldn't it be fun to build your own Google?

#38

Post by minthd »

sparkzilla wrote: post #28

The problem with algorithmic/scraper search methods, is that they only work with existing data. For example, most Google searches gives a list of websites on one side, and some data scraped from Wikipedia on the other. There is not much meaning there. That's because Google's algorithm cannot combine the results into something original, because that would require human creativity. As such, I see the rise of different…

You really discount the value of the long tail of search, which is where you get best info from google.

Re: Wouldn't it be fun to build your own Google?

#40

Post by jobposter1234 »

threeseed wrote: post #37
jobposter1234 wrote: post #35

Earlier quoted context omitted.

Isn't google doing that because it detected the semantic information was on the page, even if the exact term wasn't? Is your issue with the fact that they're doing more than just a keyword retrieval, or is your issue with the fact that they're doing it poorly?

Isn't my issue obvious ? I wanted search results that contained the search terms. Otherwise I wouldn't have entered them in the first place. I understand Google is trying to be clever here and appealing to novices who don't really understand what they want. But my point is that for those of us that do it is an incredibly annoying "feature". Feature is in quotes because in the specific case above they didn't find sema…

>Isn't my issue obvious ?

Not touching that with a 10 foot pole... (just teasing, just teasing...)

But seriously, it sounds like what you want is a keyword matching engine. Google, for better or worse, has decided they know enough about their user's searches that they don't mind modifying the query parameter in an attempt to retrieve what people want rather than what they literally say they want.

I understand that you don't feel they're serving your needs any longer, and that can be frustrating. I think, however, that you can preceded mandatory terms with a + sign to require it to be present on page.

Post reply on HN