Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Imagine a search engine that removed top million sites from its index

millionshort.com

Post reply on HN
Print view View original

41–50 of 204 posts

Re: Show HN: Imagine a search engine that removed top million sites from its index

#42

Post by taxonomyman »

g_lined wrote: post #35

This is a great idea and I see myself coming back to this. It's a shame that a little blog on tumblr or blogspot gets taken out because it's under a big name domain - but this has spam related benefits too. Great work!

Maybe we can set up a Webmaster tools sort of submission process for inclusion.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#43

Post by selectout »

taxonomyman wrote: post #34
selectout wrote: post #29

Not sure if I found an anomaly or what, but a simple search of "Privacy" returns results from thesaurus.com, merriam-webster.com, truste.com, kelloggcompany.com, and many more that are all in the top few thousand according to QuantCast and Compete. Great idea though, will definitely try this out some more.

merriam-webster.com redirects to m-w.com - looking into the others.. Sometimes if a site has header redirects it gets lost in the filters. Thx for pointing this out.

ahh that explains it. I'm actually working on a personal project right now and this could help out quite a bit, so I am excited to see where this goes. Best of luck!

Re: Show HN: Imagine a search engine that removed top million sites from its index

#44

Post by jpk »

ssharp wrote: post #25
Cushman wrote: post #16

I'd be fascinated to see the kind of SEO that would go on if this took off. "Bad news-- we're a top 100 hit for several of our main keywords. We'll have to change our URL scheme again."

High page rank pages / sites could start holding competitors' pages hostage by linking to them. Maybe we'd see a resurgence in Flash.

So what we're saying is: "This would be a terrible, terrible thing for the web."

Re: Show HN: Imagine a search engine that removed top million sites from its index

#45

Post by taxonomyman »

yayadarsh wrote: post #39

After a few test searches, this is surprisingly effective for things which I had resigned "un-findable" because of poor Google results. This is most apparent on non-technical things, in this case specific Jazz chord fingerings for a guitar class I am taking. I am very interested as to what comes of this, or rather what is influenced by its implications.

That was exactly the goal. In our opinion the "un-findable" as you put it represents a gold mine of information yet to be absorbed and enjoyed.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#47

Post by taxonomyman »

wildmXranat wrote: post #28
felixchan wrote: post #24

How did you build this? Are you indexing the entire web yourself? Or are you using Google's index/removing the top 1 million based on domain?

I think search APIs like Yahoo BOSS allow you pass arguments that contain a black list of domains. I think it's the 'sites' argument that may be used like this: &sites=-google.com

You are right, but they won't allow the list to be 1 million sites long. You are talking 15 megs of data per data in plain text per request.

But I like the idea of being able for users to, via a setting perhaps, add their own list of deny/include sites.

Thx for the comment.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#50

Post by taxonomyman »

mserdarsanli wrote: post #48

Wow that is pretty awesome. I reached some results I want that I could not find via popular search engines with hours of searching. Believe it or not, this engine is changing my life.

Glad that code is being put to good.
Post reply on HN