Show HN: Imagine a search engine that removed top million sites from its index • Hacker News Board

Re: Show HN: Imagine a search engine that removed top million sites from its index

#31

Post by pbhjpbhj » Mon, Apr 30, 2012, 6:52 PM UTC

Would you share some implementation details.

What's your source for the top million sites; where do you get your site list from for the other results?

Re: Show HN: Imagine a search engine that removed top million sites from its index

#32

Post by tnorthcutt » Mon, Apr 30, 2012, 6:53 PM UTC

And of course, W3Schools still manages to show up, thanks to their multiple crazy subdomains: http://cl.ly/GFup

Re: Show HN: Imagine a search engine that removed top million sites from its index

#33

Post by carlosaguayo » Mon, Apr 30, 2012, 6:53 PM UTC

If you search for "google" and remove the top million results, you still get google main page (in this case, the one for australia and india...)

Re: Show HN: Imagine a search engine that removed top million sites from its index

#34

Post by taxonomyman » Mon, Apr 30, 2012, 6:53 PM UTC

selectout wrote: post #29

Not sure if I found an anomaly or what, but a simple search of "Privacy" returns results from thesaurus.com, merriam-webster.com, truste.com, kelloggcompany.com, and many more that are all in the top few thousand according to QuantCast and Compete. Great idea though, will definitely try this out some more.

merriam-webster.com redirects to m-w.com - looking into the others.. Sometimes if a site has header redirects it gets lost in the filters. Thx for pointing this out.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#35

Post by g_lined » Mon, Apr 30, 2012, 6:54 PM UTC

This is a great idea and I see myself coming back to this. It's a shame that a little blog on tumblr or blogspot gets taken out because it's under a big name domain - but this has spam related benefits too.

Great work!

Re: Show HN: Imagine a search engine that removed top million sites from its index

#36

Post by dahumpty » Mon, Apr 30, 2012, 6:54 PM UTC

zenpaul wrote: post #22

duck wrote: post #14

Great idea, although I think if you could explain it a bit better you could avoid the confusion like several of these comments are showing. I like how my Hacker Newsletter project shows up #2 when searching for Hacker News. :)

And how do you determine the "top" million sites?

Check out:

http://www.alexa.com/topsites

Re: Show HN: Imagine a search engine that removed top million sites from its index

#37

Post by eliasmacpherson » Mon, Apr 30, 2012, 6:54 PM UTC

This is a breath of fresh air - I'm loving the unpredictability of the top results! It's like flicking through a new set of 1000 tv channels in a different country.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#38

Post by taxonomyman » Mon, Apr 30, 2012, 6:55 PM UTC

tnorthcutt wrote: post #32

And of course, W3Schools still manages to show up, thanks to their multiple crazy subdomains: http://cl.ly/GFup

Your're right. We strip www from the domains. We need a better function to rip out the domain from a URL. Just haven't had time to cook one up yet.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#39

Post by yayadarsh » Mon, Apr 30, 2012, 6:56 PM UTC

After a few test searches, this is surprisingly effective for things which I had resigned "un-findable" because of poor Google results. This is most apparent on non-technical things, in this case specific Jazz chord fingerings for a guitar class I am taking.

I am very interested as to what comes of this, or rather what is influenced by its implications.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#40

Post by taxonomyman » Mon, Apr 30, 2012, 6:56 PM UTC

eliasmacpherson wrote: post #37

This is a breath of fresh air - I'm loving the unpredictability of the top results! It's like flicking through a new set of 1000 tv channels in a different country.

The positive feedback is awesome.. Thx