Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wouldn't it be fun to build your own Google?

radar.oreilly.com

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Wouldn't it be fun to build your own Google?

#21

Post by dugmartin »

andrewhillman wrote: post #14
dugmartin wrote: post #12

Earlier quoted context omitted.

As a site owner you can pay Google for this: https://www.google.com/work/search/products/gss.html I used it several years ago to power this site search: http://www.poetryfoundation.org/search/articles#qs=ginsberg It works pretty well (disclaimer: I'm not sure this is still being used on that site)

Yes. It's been around for years. I have used it but this misses my point or proves it. Either way, sites need to figure out how to power own search.

This is customized for the site to 'power their own search'. The query sent to Google is based on the keyword but also on the article type (in this example) and then the xml that is returned is post-processed to have a custom render per article type.

Since we had all the data the results from Google were compared to a local search I setup in Sphinx (http://sphinxsearch.com/) and the Google results were more relevant and it was a lot cheaper to deploy.

Re: Wouldn't it be fun to build your own Google?

#22

Post by runarb »

Arkanosis wrote: post #19
mjklin wrote: post #4

I thought Wikimedia tried this once. Big announcement, then nothing. Is that code still available?

That was Wikia, not Wikimedia, but yes, the code is still available: - crawler: http://sourceforge.net/projects/grub/ - search engine: http://nutch.apache.org/

If my memory serves me correctly so is it only the client part of grub that is open source. Without the server part on cannot use it to setup one’s own crawl.

Re: Wouldn't it be fun to build your own Google?

#23

Post by Animats »

angersock wrote: post #15

For anyone interested, there's a hilariously bitter and practical paper on the trials and tribulations of building a search engine: http://queue.acm.org/detail.cfm?id=988407 EDIT: Article is clearly from an earlier era, but it's really cool to see how far we've come and how much more computing power we have available now. There are entire categories of problems that simply don't exist anymore.

She later designed the search engine of Cuil. While Cuil failed, it only cost them about $30 million to do most of what Google does.

It's surprising to me that there aren't search engines from Comcast, AT&T, and Apple. If you have customers, why give up all that ad revenue to Google? Google is paying some big players a lot of money not to do that. They were paying Apple $1 billion a year to be the default on Apple products. Apple switched from Google to Bing anyway.

Re: Wouldn't it be fun to build your own Google?

#25

Post by pasteurquadrant »

curiously wrote: post #20
Smerity wrote: post #10

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it…

what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their human uploaded content and profiles, and majority of the data from the web being useless without a context and purpose of the searcher? I'm just curious as to how commoncrawl compares with kimonolabs and import.io as they seem to have the same goal of creating…

+1

Re: Wouldn't it be fun to build your own Google?

#26

Post by arthurcolle »

andrewhillman wrote: post #9
arthurcolle wrote: post #6

Earlier quoted context omitted.

I'm pretty sure Elastic is on the right direction in this regard, don't you think?

I don't think so. I just went to their site and the first case study i opened was from theguardian.com. I went to theguardian.com and went to do a search. Guess who they are using to power their search function? Google. In my opinion which means nothing, sites need to figure out how to power their own search. Using a third party isn't going to work for most. Maybe people need to focus on building custom architecture…

Maybe I'm beating a dead horse but I feel like you could create a pretty compelling search engine using ES. I used it this summer at Goldman and it seems like it will really change the landscape... It's just so fast. I mean full-text-indexing seems to be a pretty integral part of search. Maybe it's just the first step, with the second step being a well written ranking function, but that's just my thought.

Re: Wouldn't it be fun to build your own Google?

#27

Post by jtoy »

Smerity wrote: post #10

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it…

anyway we can get an open sourced version of that optimized job for us to play with?

Re: Wouldn't it be fun to build your own Google?

#28

Post by sparkzilla »

The problem with algorithmic/scraper search methods, is that they only work with existing data. For example, most Google searches gives a list of websites on one side, and some data scraped from Wikipedia on the other. There is not much meaning there. That's because Google's algorithm cannot combine the results into something original, because that would require human creativity. As such, I see the rise of different kinds of search based on what humans create, rather than what computers can scrape. I wrote a (longish) blog post on this problem: http://newslines.org/blog/googles-black-hole/

Re: Wouldn't it be fun to build your own Google?

#29

Post by threeseed »

discardorama wrote: post #16

Google's power comes not from the crawling, but from the retrieval and ranking. They use many more signals than the hyperlinks and anchor text (which is all you'd have if you crawled yourself). Indexing crawled content would have been OK in the year 2000; but today, the users demand more. Relevance is the top priority, and no one does it better than El Goog.

Sorry but Google's ranking algorithm for me is far from brilliant.

To give you an example, search for "webhcat primary key" (without quotes) and note how the top three search results do not actually contain the term webhcat. Google constantly does this. It randomly ignores search terms unless you explicitly quote them.

I believe that there is still a market for a technical/advanced search engine.

Re: Wouldn't it be fun to build your own Google?

#30

Post by dredmorbius »

discardorama wrote: post #16

Google's power comes not from the crawling, but from the retrieval and ranking. They use many more signals than the hyperlinks and anchor text (which is all you'd have if you crawled yourself). Indexing crawled content would have been OK in the year 2000; but today, the users demand more. Relevance is the top priority, and no one does it better than El Goog.

Also on their dominance of the advertising market from which the value of crawling comes.
Post reply on HN