Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Algorithmic search is sinking

skrenta.com

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Algorithmic search is sinking

#31

Post by tgflynn »

ddemchuk wrote: post #24
mixmax wrote: post #7

Or maybe our algorithms just aren't good enough. Suppose you use bayesian filtering on the text surrounding the links to determine whether the connection is good or bad. With a reasonable amount of data it should be possible. Note: I'm not an algorithms guy, I do business and strategy and a wee bit of programming, so maybe the example isn't good, but I thinkthe point is.

Google already analyzes backlinks in their context to determine how relevant the anchor text is to the topic of the page. Determining sentiment (the topic of the NYT piece) is considerably harder though, because it would allow for spammers to write negative articles about a site and link to it and negatively affect its rankings. Also, determining the tone/emotions of a piece of text is probably one of the hardest thi…

You wouldn't have to apply the sentiment factor for all sites. You could have a rule that would only lower rankings for sites which have an overwhelming majority of negative links (say >90%). I don't think there would be many cases where a spammer could achieve that level of impact and those few cases could be dealt with through manual intervention.

As for determining sentiment, it's not something I've ever tried to do, but is it really that hard ? Intuitively I would think positive and negative articles would have significantly different distributions of certain words.

Re: Algorithmic search is sinking

#32

Post by evgen »

jeremydavid wrote: post #17

"The only way to combat this and return trust and quality to search is by taking an editorial stand and having humans identify the best sites for every category." There are billions of webpages. Who is going to do this review? Is someone honestly going to review http://stackoverflow.com/questions/4300234/how-might-union-f... and put it in the category of "How Union/Find data structures can be applied to Kruskal's alg…

The closest thing to a editorialized web is www.dmoz.org, and that hasn't been properly updated in years (and never will be) because it failed.

And noting circular irony one often sees, Rich Skrenta created a Yahoo knock-off in the bubble days called NewHoo and then sold it to Mozilla where it became the seed of dmoz...

Re: Algorithmic search is sinking

#33

Post by btilly »

evgen wrote: post #32
jeremydavid wrote: post #17

"The only way to combat this and return trust and quality to search is by taking an editorial stand and having humans identify the best sites for every category." There are billions of webpages. Who is going to do this review? Is someone honestly going to review http://stackoverflow.com/questions/4300234/how-might-union-f... and put it in the category of "How Union/Find data structures can be applied to Kruskal's alg…

The closest thing to a editorialized web is www.dmoz.org, and that hasn't been properly updated in years (and never will be) because it failed. And noting circular irony one often sees, Rich Skrenta created a Yahoo knock-off in the bubble days called NewHoo and then sold it to Mozilla where it became the seed of dmoz...

Well then he has been at this for a lot of years, and perhaps now knows how to do it. :-)

Re: Algorithmic search is sinking

#34

Post by prodigal_erik »

wallflower wrote: post #11
McKittrick wrote: post #9

Earlier quoted context omitted.

yes, this is all getsatisfaction's fault. surprised nytimes missed that angle.

Their response: http://blog.getsatisfaction.com/2010/11/28/when-businesses-a...

see also http://news.ycombinator.com/item?id=1948934

Re: Algorithmic search is sinking

#35

Post by adambyrtek »

Vivtek wrote: post #2

I have heard about search engine spam before and sort of discounted it - but you know, if you search on something that's not a technical topic or something equally specific like a band name, that is, you're searching on a general topic that is of interest to the mundanes, then there really is a whole lot of spam on Google. My case from this week was that I wanted plans for a bookcase. I searched, therefore, on "build…

Google is all about search quality, so they have a huge motivation to fight spam. It's getting worse now because the mail spam is mostly a solved problem[1] and the web has became the new battlefront.

[1] I haven't got a single spam in my Gmail inbox for ages, and other spam filters are pretty good as well.

Re: Algorithmic search is sinking

#37

Post by reitzensteinm »

cosgroveb wrote: post #22
sandGorgon wrote: post #20

Earlier quoted context omitted.

I invoke SandGorgon’s law of outsourcing analogies As an online discussion about PROGRAMMING grows longer, the probability of a comparison involving outsourcing or Indians approaches 1, if Godwin’s law has not already been satisfied

I really don't see that many comments about Indians and outsourcing in programming discussions here. Or maybe I am not noticing them?

Also, the law is pretty much a tautology since it doesn't define a time scale... eventually, pretty much everything gets said.

"As an online discussion about PROGRAMMING grows longer, the probability of a discussion of traditional medicine and spiritual beliefs surrounding childbirth in ancient sub Saharan African tribes approaches 1."

Re: Algorithmic search is sinking

#38

Post by sandGorgon »

reitzensteinm wrote: post #37
cosgroveb wrote: post #22

Earlier quoted context omitted.

I really don't see that many comments about Indians and outsourcing in programming discussions here. Or maybe I am not noticing them?

Also, the law is pretty much a tautology since it doesn't define a time scale... eventually, pretty much everything gets said. "As an online discussion about PROGRAMMING grows longer, the probability of a discussion of traditional medicine and spiritual beliefs surrounding childbirth in ancient sub Saharan African tribes approaches 1."

AS Godwin's law itself clarifies:

Godwin put forth the sarcastic observation that, given enough time, all discussions—regardless of topic or scope—inevitably end up being about Hitler and the Nazis.

Re: Algorithmic search is sinking

#39

Post by viraptor »

Vivtek wrote: post #2

I have heard about search engine spam before and sort of discounted it - but you know, if you search on something that's not a technical topic or something equally specific like a band name, that is, you're searching on a general topic that is of interest to the mundanes, then there really is a whole lot of spam on Google. My case from this week was that I wanted plans for a bookcase. I searched, therefore, on "build…

Another interesting example is trying to look for a hotel. Try to search for the one you know - there's ~0 chance that their actual page will appear on the front page. Only resellers, spam and car renting...

Re: Algorithmic search is sinking

#40

Post by smackay »

TheCoreh wrote: post #3
Vivtek wrote: post #2

I have heard about search engine spam before and sort of discounted it - but you know, if you search on something that's not a technical topic or something equally specific like a band name, that is, you're searching on a general topic that is of interest to the mundanes, then there really is a whole lot of spam on Google. My case from this week was that I wanted plans for a bookcase. I searched, therefore, on "build…

That's strange. My Google results page for "build a bookcase" shows 8 high quality tutorials on how to build bookcases, besides two pretty decent video-tutorials.

I concur. I got to the second page before I found anything other than a first class result (incidently it was how to build a bookcase in 5 minutes which seems to be a dubious proposition at best). Now if you looking for information on something you actually want to buy then yes, google's results are indeed a sea of spam.
Post reply on HN