Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wouldn't it be fun to build your own Google?

radar.oreilly.com

Post reply on HN
Print view View original

11–20 of 63 posts

Re: Wouldn't it be fun to build your own Google?

#11

Post by nemothekid »

arthurcolle wrote: post #6
andrewhillman wrote: post #5

Yeah, this sounds all well and good in theory, but after visiting thousands of sites over the years, it might be a better idea to help engineers build a search engine for their own site/data first. I can't recall many websites that have amazing search. It's a problem when I have to use google to find what I want on xyz.com because if I go search for what I am looking for on xyz.com I cannot find it even if I know its…

I'm pretty sure Elastic is on the right direction in this regard, don't you think?

I think "full-text indexing" is a very different problem than "search."

Full-text indexing (what ES provides) has been around for almost forever, ES just does a way better job of productizing/delivering it.

However, Google is far more than a text index. Ranking, currently is still very difficult and requires messing around with facet and weighting parameters.

Re: Wouldn't it be fun to build your own Google?

#12

Post by dugmartin »

andrewhillman wrote: post #5

Yeah, this sounds all well and good in theory, but after visiting thousands of sites over the years, it might be a better idea to help engineers build a search engine for their own site/data first. I can't recall many websites that have amazing search. It's a problem when I have to use google to find what I want on xyz.com because if I go search for what I am looking for on xyz.com I cannot find it even if I know its…

As a site owner you can pay Google for this:

https://www.google.com/work/search/products/gss.html

I used it several years ago to power this site search:

http://www.poetryfoundation.org/search/articles#qs=ginsberg

It works pretty well (disclaimer: I'm not sure this is still being used on that site)

Re: Wouldn't it be fun to build your own Google?

#13

Post by JDDunn9 »

I've always wanted to experiment with my own search algorithm. Unfortunately, I think this is still out of the budget of average programmers. Just the hard drives to download 1.3 petabytes would cost six-figures.[1][2]

[1] https://www.backblaze.com/petabytes-on-a-budget-how-to-build...

[2] https://www.backblaze.com/blog/why-now-is-the-time-for-backb...

Re: Wouldn't it be fun to build your own Google?

#14

Post by andrewhillman »

dugmartin wrote: post #12
andrewhillman wrote: post #5

Yeah, this sounds all well and good in theory, but after visiting thousands of sites over the years, it might be a better idea to help engineers build a search engine for their own site/data first. I can't recall many websites that have amazing search. It's a problem when I have to use google to find what I want on xyz.com because if I go search for what I am looking for on xyz.com I cannot find it even if I know its…

As a site owner you can pay Google for this: https://www.google.com/work/search/products/gss.html I used it several years ago to power this site search: http://www.poetryfoundation.org/search/articles#qs=ginsberg It works pretty well (disclaimer: I'm not sure this is still being used on that site)

Yes. It's been around for years. I have used it but this misses my point or proves it. Either way, sites need to figure out how to power own search.

Re: Wouldn't it be fun to build your own Google?

#15

Post by angersock »

For anyone interested, there's a hilariously bitter and practical paper on the trials and tribulations of building a search engine:

http://queue.acm.org/detail.cfm?id=988407

EDIT:

Article is clearly from an earlier era, but it's really cool to see how far we've come and how much more computing power we have available now. There are entire categories of problems that simply don't exist anymore.

Re: Wouldn't it be fun to build your own Google?

#16

Post by discardorama »

Google's power comes not from the crawling, but from the retrieval and ranking. They use many more signals than the hyperlinks and anchor text (which is all you'd have if you crawled yourself). Indexing crawled content would have been OK in the year 2000; but today, the users demand more. Relevance is the top priority, and no one does it better than El Goog.

Re: Wouldn't it be fun to build your own Google?

#18

Post by marktangotango »

Smerity wrote: post #10

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it…

Before Google's page rank algorithm there was a lot of research into document search. A favorite of mine was 'scatter gather'[1].

I often wanted ability to filter and group when staring a pages of results from redhat jboss support forums when trying to fix a dead jboss cluster. But I quit that job so haven't had the need recently.

Edit; point being it would be nice if someone came up with a service that implemented the ideas in this paper :)

[1] http://www-users.cs.umn.edu/~han/dmclass/scatter.pdf

Re: Wouldn't it be fun to build your own Google?

#19

Post by Arkanosis »

mjklin wrote: post #4

I thought Wikimedia tried this once. Big announcement, then nothing. Is that code still available?

That was Wikia, not Wikimedia, but yes, the code is still available: - crawler: http://sourceforge.net/projects/grub/ - search engine: http://nutch.apache.org/

Re: Wouldn't it be fun to build your own Google?

#20

Post by curiously »

Smerity wrote: post #10

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it…

what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their human uploaded content and profiles, and majority of the data from the web being useless without a context and purpose of the searcher?

I'm just curious as to how commoncrawl compares with kimonolabs and import.io as they seem to have the same goal of creating an internet as a dataset, or an API. I can't help but feel like it's just solving another 'semantic web' problem that nobody asked for.

It is funny that the most demanding customers of semantic web are also the ones who are willing to pay the least amount of time and money.

Post reply on HN