Show HN: Imagine a search engine that removed top million sites from its index • Hacker News Board

Re: Show HN: Imagine a search engine that removed top million sites from its index

#141

Post by robrenaud » Mon, Apr 30, 2012, 10:25 PM UTC

bane wrote: post #116

It's a similar measure that's often used in NLP. Sentences, documents etc. are usually stripped of common or popular terms first and the remaining ones tend to have higher information value. It's not entirely a surprise that it works for meta-language constructs like the web and site popularity.

Uhh,

1) I am basically certain that it doesn't work. Imagine something this simple actually did work in general. Do you think Google, Bing, etc, wouldn't have implemented it?

2) I think the analogy is extremely flawed. nytimes.com is by no means a web page equivalent to "the" or "a" in language. Articles, pronouns, etc, don't really carry meaning. Despite being popular, nytimes certainly does.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#142

Post by PBxdId7wFr » Mon, Apr 30, 2012, 10:29 PM UTC

afhof wrote: post #136

nostromo wrote: post #102

It's interesting to see popularity used as an inverse corollary with quality. Imagine a TV that skipped the most popular programming (goodbye American Idol), or a radio station that only plays non-hits. Of course, there are great websites out there that are very popular (Wikipedia, NYTimes/WSJ, StackOverflow). I'd love to see a search engine with a better signal for quality than non-popularity (this search engine), o…

You have to be careful with that idea since it can still lead to poor quality. My campus's radio station (WREK Atlanta) seems to play just weird music for the sake of being weird. They never play popular music, but they rarely play any good music either.

Or you may have different tastes from the DJs.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#143

Post by mburshteyn » Mon, Apr 30, 2012, 10:37 PM UTC

taxonomyman wrote: post #23

waivej wrote: post #21

Wow... It felt like using Google 10 years ago. I think you are onto something.

Wow.. awesome comliment..

Usually don't search unless I'm looking for something in particular, but just played with this for a good 15 minutes running random queries. The results are really good and at the same time I'm discovering sites I'd never otherwise see with Google.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#144

Post by Ben_Burke » Mon, Apr 30, 2012, 10:47 PM UTC

i searched for my site and in teh goog i get first page....here i found nothing....so for me this = no good. I understand the base but i dont understand the result

Re: Show HN: Imagine a search engine that removed top million sites from its index

#145

Post by benn » Mon, Apr 30, 2012, 10:48 PM UTC

How does this work? Are you using another search engines API or did you crawl the web yourself?

Re: Show HN: Imagine a search engine that removed top million sites from its index

#146

Post by halle_lu_jah » Mon, Apr 30, 2012, 10:49 PM UTC

Here's what Dropbox thinks about power users:

WSJ: What's next?

Mr. Ferdowsi: We continue to focus on actually solving problems that real people have and not being distracted by what power users want.

Google has made clear what they think about power users:

No + operators in search.

No web-based code search.

No Google Labs for the public.

etc.

Plenty of wood behind the Google arrows, but all the cool ones have been cast out of the quiver.

Just what kind of targets is Google aiming at nowadays?

Millionshort I give you +999,999.

I would give you +1M if you took out the AdSense and PlusOne javascript.

This has been a long time coming.

Alas, DDG and other alternatives are all about _money_.

Search is about _discovery_.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#147

Post by dools » Mon, Apr 30, 2012, 10:53 PM UTC

pjin wrote: post #123

Funny, on my first query I found an obscure HN scraper: http://tazod.com/

Wow I think this is the exact idea I had for "HN Time Machine" - basically a thing to extend the life of the newest page and speed up the movement of stuff through the home page (ie. I think the HN new page moves too quickly and the home page not quickly enough)

Re: Show HN: Imagine a search engine that removed top million sites from its index

#148

Post by doktrin » Mon, Apr 30, 2012, 11:02 PM UTC

I'm getting odd results with the following query :

Search String : Ruby

Remove From Top : 1000 & 10000

In both instances, the top hit is http://www.ruby-lang.org, which is also the top hit from both Google and DDG.

Am I missing something?

edit: formatting

Re: Show HN: Imagine a search engine that removed top million sites from its index

#149

Post by Tossrock » Mon, Apr 30, 2012, 11:08 PM UTC

Well, in a rather meta turn of events, searching for my username on this returned a link to hackerbra.in, which appears to be some kind of HN mirror.

Re: Show HN: Imagine a search engine that removed top million sites from its index

#150

Post by ricardobeat » Mon, Apr 30, 2012, 11:14 PM UTC

nostromo wrote: post #102

It's interesting to see popularity used as an inverse corollary with quality. Imagine a TV that skipped the most popular programming (goodbye American Idol), or a radio station that only plays non-hits. Of course, there are great websites out there that are very popular (Wikipedia, NYTimes/WSJ, StackOverflow). I'd love to see a search engine with a better signal for quality than non-popularity (this search engine), o…

Youtube, grooveshark and others kind of fill that gap. They still promote popular content though, but that's unavoidable unless you take money out of the equation.

http://channel101.com/ offers a nice selection of "unpopular" shows. You should check Danny Jelinek's "Everything" (https://vimeo.com/channels/231109) if you're into video-art.