Live data from Hacker News

Algorithmic search is sinking

skrenta.com

21–30 of 49 posts

Re: Algorithmic search is sinking

#21
post #19

So there's some shitty retailer. What does this have to do with algorithmic search?

Well, Google uses algorithms to rank websites; the retailers unprofessional practices are gaming Google's algorithms by increasing the number of inbound links pointing to their site, which increases their page rank on a search return on Google and increases traffic.

Re: Algorithmic search is sinking

#22

I don't know if Skrenta's approach is perfect (can spammers make slashtags? I'll bet they can!) but Google's is clearly failing. Giant swathes of Google searches are now overrun with datafog spammers. Ehow, squidoo, hubpages, wikihow, buzzle, how-wiki, ezinearticles, bukisa, wisegeek, articlesnatch, healthblurbs, associatedcontent - all thee and thousands more domains filled with spam semi-automatically generated by…

I invoke SandGorgon’s law of outsourcing analogies As an online discussion about PROGRAMMING grows longer, the probability of a comparison involving outsourcing or Indians approaches 1, if Godwin’s law has not already been satisfied

I really don't see that many comments about Indians and outsourcing in programming discussions here. Or maybe I am not noticing them?

Re: Algorithmic search is sinking

#23
post #15

There is little rigor behind most of the claims of the NYTime story: The targeted site already negates any pagerank benefit of their links (they do implement nofollow), and the definitive example seems to be nothing more than good SEO of the site in question (most of the other front and second page sites are pretty mediocre as well, clearly with little web competition in the keyword space). In any case, go to a shopp…

Note: I do SEO as part of my job, so I know a few tools that can look through this. Google keyword checker gives 590 searches a month for that phrase, so it's not too competitive. I'm sure he ranks for a lot of these tail phrases though.

A lot of his juice comes from every page (seems to be over 10,000 according to Yahoo Site Explorer) on his site linking with good anchor text to every other page. The fact that he ranks so low (on my Google he's number 6 or so) even with this on such an easy term shows something, doesn't it?

Re: Algorithmic search is sinking

#24
post #7

Or maybe our algorithms just aren't good enough. Suppose you use bayesian filtering on the text surrounding the links to determine whether the connection is good or bad. With a reasonable amount of data it should be possible. Note: I'm not an algorithms guy, I do business and strategy and a wee bit of programming, so maybe the example isn't good, but I thinkthe point is.

Google already analyzes backlinks in their context to determine how relevant the anchor text is to the topic of the page.

Determining sentiment (the topic of the NYT piece) is considerably harder though, because it would allow for spammers to write negative articles about a site and link to it and negatively affect its rankings. Also, determining the tone/emotions of a piece of text is probably one of the hardest things to do with textual analysis

Re: Algorithmic search is sinking

#25
The mystery of the PageRank algorithm is not only a defense against gaming, it's a defense against competition. Other than stylistic differences (a la Bing), it seems difficult to differentiate a new service when nobody understands the details of the standard one.

As a net addict, I regularly find myself frustrated because I can't figure out how to get meaningful information out of Google instead of sites trying to sell me. And if I can't think off the top of my head of a website that will act as a relevant portal for that kind of info, then there isn't really any alternative to Google.

At least, not that I know of yet: can anyone suggest one?

Google has done amazing things for our ability to get what we want and fast, but it also is slowly eroding our independence from it and our ability to educate ourselves by other means.

Here's hoping they prove worthy stewards once they own all the information on the planet.

Re: Algorithmic search is sinking

#26
post #7

Or maybe our algorithms just aren't good enough. Suppose you use bayesian filtering on the text surrounding the links to determine whether the connection is good or bad. With a reasonable amount of data it should be possible. Note: I'm not an algorithms guy, I do business and strategy and a wee bit of programming, so maybe the example isn't good, but I thinkthe point is.

Interesting, do people frequently use bayes' theorem in web programming? Ive only seen it it other programming contexts.

If you're interested, there are a whole host of fun and useful machine learning techniques that are actually not as hard to understand and apply as they sound. The best introductory book that I know of is Programming Collective Intelligence, which is surprisingly clear, if a little vague on the theory:

http://oreilly.com/catalog/9780596529321

Naive Bayesian classifiers are just one of the more popular types; others include Support Vector Machines (SVMs), decision trees (and their relatives, random forests), and a bunch more. If you'd like to play around with some, Weka is good open source software for this:

http://www.cs.waikato.ac.nz/ml/weka/

Re: Algorithmic search is sinking

#27
post #16

it is sinking, but for a different reason. the web is getting away from google. getting locked up in apps, or walled gardens like facebook or itunes

This. The open content web is beginning to disintegrate, being displaced by siloed apps which only incidentally happen to involve HTML and HTTP.

Re: Algorithmic search is sinking

#28
post #7

Or maybe our algorithms just aren't good enough. Suppose you use bayesian filtering on the text surrounding the links to determine whether the connection is good or bad. With a reasonable amount of data it should be possible. Note: I'm not an algorithms guy, I do business and strategy and a wee bit of programming, so maybe the example isn't good, but I thinkthe point is.

Google already analyzes backlinks in their context to determine how relevant the anchor text is to the topic of the page. Determining sentiment (the topic of the NYT piece) is considerably harder though, because it would allow for spammers to write negative articles about a site and link to it and negatively affect its rankings. Also, determining the tone/emotions of a piece of text is probably one of the hardest thi…

Determining sentiment (the topic of the NYT piece) is considerably harder though, because it would allow for spammers to write negative articles about a site and link to it and negatively affect its rankings.

This could be solved by making sentiments act as a weight (i.e. a multiplier in [0, 1]). Positive sentiments would give a particular reference more weight, negative sentiments would give little to no weight. Then it would be impossible to negatively affect a site's rankings - only positively affect them. Just like now.

Re: Algorithmic search is sinking

#29
post #7

Or maybe our algorithms just aren't good enough. Suppose you use bayesian filtering on the text surrounding the links to determine whether the connection is good or bad. With a reasonable amount of data it should be possible. Note: I'm not an algorithms guy, I do business and strategy and a wee bit of programming, so maybe the example isn't good, but I thinkthe point is.

Google already analyzes backlinks in their context to determine how relevant the anchor text is to the topic of the page. Determining sentiment (the topic of the NYT piece) is considerably harder though, because it would allow for spammers to write negative articles about a site and link to it and negatively affect its rankings. Also, determining the tone/emotions of a piece of text is probably one of the hardest thi…

A negative link on a page with a low reputation could be accounted for accordingly.

Re: Algorithmic search is sinking

#30
An awful lot seems to be getting made out of this one story, and there's really precious little else cited in the post other than generic claims of gloom and doom about search. Google's been fighting spam sites for a long time before this and the battle certainly waxes and wanes but I'm sceptical that it's actually being lost, it's just a constant struggle.

Now if you tell me that there is value in social search we could have a totally different discussion, but it's more about the persuasive power of personal recommendation than algorithms not working any more.

Post reply on HN