“The world's most private search engine”

Post by rdl » Fri, Jun 21, 2013, 1:25 PM UTC

ChuckMcM wrote: post #10

rdl wrote: post #7

I wonder what it takes to host a full search engine for the Internet now, if an organization wanted genuinely private search. Presumably you can buy crawls from somewhere, but does anyone sell a self-hostable general web search engine?

That I'm aware of there are only three places to get actual search index results in the US, Google, Bing (via Yahoo BOSS), and of course Blekko. You can also get index results from both Yandex (Russian) and Baidu (Chinese) but both serve them out of data centers in either Amsterdam or Asia respectively so latency is a bit longer. With modern hardware (especially 3TB drives) you can crawl a significant chunk of the In…

I guess it would make more sense to let an organization specify sets of interest. Optionally fail-through to external public search engines if necessary.

e.g. if I were an intel agency, I'd crawl all the jihadi sites and build a special translation/search tool for those (I'm sure they've done this). No need to crawl the rest of the pornosphere, so it could be done on one box. A commercial company could search all their industry sites locally, and then fail-through for general stuff.

I guess my main question is "how many machines are needed to do 1-3 term search on a 0.005-0.5 billion URL index" for one user with decent latency, and then how many simultaneous users could be served from that, after the index is delivered". The idea being that organizations will trust the security guarantees of a machine they own, so you could build a search service in a way that the crawl/etc. is done by one set of shared machines, bulk data streamed to everyone inside the datacenter, certain data (up to the entirety) retained by each end-user organization's query cluster, and then queries done by that cluster locally to keep search terms secret.

I assume the actual query part could fit on single machines -- you would sort of want to keep cached pages for a lot of the things you'd want to keep searches secret for, but disk is cheap.

Post by ChuckMcM » Fri, Jun 21, 2013, 3:08 PM UTC

rdl wrote: post #21

ChuckMcM wrote: post #10

Earlier quoted context omitted.

That I'm aware of there are only three places to get actual search index results in the US, Google, Bing (via Yahoo BOSS), and of course Blekko. You can also get index results from both Yandex (Russian) and Baidu (Chinese) but both serve them out of data centers in either Amsterdam or Asia respectively so latency is a bit longer. With modern hardware (especially 3TB drives) you can crawl a significant chunk of the In…

I guess it would make more sense to let an organization specify sets of interest. Optionally fail-through to external public search engines if necessary. e.g. if I were an intel agency, I'd crawl all the jihadi sites and build a special translation/search tool for those (I'm sure they've done this). No need to crawl the rest of the pornosphere, so it could be done on one box. A commercial company could search all the…

In that particular case it gets a lot easier of course. One of the challenges of "general" search is that you're trying to satisfy a wide variety of queries, for domain specific search you can be much more selective of what you pick. Also one of the 'knobs' if you will is change in the corpus, so if you index Wikipedia for example it doesn't change all that much and that is a pretty stable index, versus if you index say a few tens of thousands of blogs then they will get new articles to index once every .5 to maybe 3 days. Versus keeping an eye on the "news" sites which get perhaps a dozen articles a day added to them.

Then if you don't keep the original extractions around, just holding the index can be done on a relatively small number of machines (say 5 or 10) and for a small user population that is pretty doable.

Google sold its "Google Appliance" which did this for businesses and often that was less than 1 cabinet or rack worth of machines.

Given some of the silly things people try to scrape Blekko for (clearly domain specific things) I'm surprised there aren't more folks building such engines. It isn't hard to write a basic crawler, it takes a lot of coaching to get it through the initial crawl since if you just blindly follow links around you can end up in endless loops on some sites or in a quasi form submission loop (we call those Crawler traps, for the most part unintentional on the part of web sites [1])

And of course we have been supporting the Common Crawl efforts to make this more generally available. We gave them a bunch of indexing data that made the Crawl they had a lot more useful as an example. So at some point we may reach critical mass and get that going.

[1] Sometimes though folks use an intentional crawler trap to find 'stealth' scrapers, which are folks making their own crawlers and not respecting robots.txt.

Re: “The world's most private search engine”

Re: “The world's most private search engine”