Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Blekko already has a list of domains to block from results

blekko.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Blekko already has a list of domains to block from results

#31

Post by seabee »

JamesDB wrote: post #26
robinduckett wrote: post #24

I'm seeing: http://www.deadmau5.com -> Really, spam? http://www.comparethemarket.com -> Price comparison site with no known biases in the UK and not owned by British Telecommunications, as it is listed. Even ONE false positive is enough to make me think this listing is a load of bullshit. How is geocities' "closing down" page spam?

A site in the UK that gets a huge amount of traffic and is backed by a large off-line national advertising campaign. Why has this been banned? Never seen spam on there, nor have seen them spamming.

Don't be fooled, that advertising campaign only used a 'sensible', similarly-named insurance business to introduce an unusual niche company[1] to the market. It's a clever ploy, you can read about it in Aleksandr Orlov's book that came out a month ago.

[1] http://www.comparethemeerkat.com/

Re: Blekko already has a list of domains to block from results

#32

Post by biot »

wybo wrote: post #18

I have been using Blekko as my primary search-engine for a couple of days now, and in my experience their search-results are very decent. Not, maybe in terms of falsely blocked sites, but certainly in terms of having fewer false positives (e.g. spam/useless pages) in the search results. Mom & pop users (and even more advanced searchers, such as students looking for book-reviews, or torrents) might very well forgive t…

> Not, maybe in terms of falsely blocked sites, but certainly in terms of having fewer false positives (e.g. spam/useless pages) in the search results.

A false positive is when a good site is mistakenly identified as a spammy site.

Re: Blekko already has a list of domains to block from results

#34

Post by joshma »

chrismealy wrote: post #2

They're not passing the "best refrigerator" test yet: http://blekko.com/ws/best+refrigerator

What exactly is the best refrigerator test? Searching for it on other sites (Google, Wikipedia) just return refrigerator results in the former and nothing really useful in the latter...

Re: Blekko already has a list of domains to block from results

#35

Post by greglindahl »

veb wrote: post #17

Wouldn't it be better to let people do their own blacklists, and then incorporate that into their official list if a percentage of people have that site down as spam?

Blekko has individual blacklists -- that's what the "spam" button by every result is.

This topspam list is algorithmic, and is not affected by user "spam" clicks.

Re: Blekko already has a list of domains to block from results

#36

Post by duskwuff »

aaronbrethorst wrote: post #13

Wow, I'm seeing tons of false positives. Why on earth is the co-written blog of a nobel laureate and a 7th circuit appeals court judge making blekko's bayesian classifier freak out? #88 - http://www.becker-posner-blog.com - bayes (spam 31.6 > 5.3)

The first item on the list currently which is not some sort of false positive appears to be wareseeker.com at #10 (and that's debatable).

Re: Blekko already has a list of domains to block from results

#37

Post by greglindahl »

* This list is generated by our algorithm, it's the most important sites which our algorithm thinks are spam. The point of making the list public is so that you guys can tell us when we're wrong. Google has a list like this, but they don't show it to anyone. Transparency in action.

Thank you all for pointing out false positives in the list. That is what we hoped would happen.

* The "nocrawl" sites are human-picked by us. Geocities is on the list because it was a very spammy domain. Even though they've (finally) removed the data, we still have old data indexed, and will remove them from the spam list once all that old data ages out.

* BT is the hosting company for comparethemarket.com

Re: Blekko already has a list of domains to block from results

#38

Post by zitterbewegung »

From all the comments and what I have noticed it sounds like a good question is "Is it better to have false positives or false negatives" in the spam problem. I personally think that its better to have false negatives then positives and a lot of the comments here seem to reflect that.

Re: Blekko already has a list of domains to block from results

#39

Post by greglindahl »

zitterbewegung wrote: post #38

From all the comments and what I have noticed it sounds like a good question is "Is it better to have false positives or false negatives" in the spam problem. I personally think that its better to have false negatives then positives and a lot of the comments here seem to reflect that.

That depends on how many false positives and negatives we have -- you don't really know how many of either. There are 100 million hosts in our crawl, how can you estimate if we have a false positive problem from looking at the list of the top 100 marked spam?

Re: Blekko already has a list of domains to block from results

#40

Post by greglindahl »

benbeltran wrote: post #7

Two big problems with automated spam blocking are: false positives and changing domain names. For the second one, how often do you revise your blocked links? what if it changed owner and the new one doesn't provide spam. For the first one, is even one false positive tolerable? Will you deny someone presence in your index because you failed? And if so, how do you handle challenges?

We don't mark a domain as spam until many of the pages we've seen look spammy.

Our ideal is to recrawl everything every 14 days, but during our launch we have not been achieving that.

Post reply on HN