Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How rel=nofollow Works

luigimontanez.com

Post reply on HN
Print view View original

11–20 of 52 posts

Re: How rel=nofollow Works

#11

Post by Terretta »

niyazpk wrote: post #9

(I am new to this debate and so may be way off, but) As far I understand, even this article is not very clear. It says: Google is simply complying with Twitter.com’s directive to not follow outbound links in tweets it crawls, and the consequence is that there will never be ”… shared this on Twitter” in the search results. Wrong. You see, when I share some random link in my blog and then you search for that topic, Goo…

> new to this ... may be way off

You are indeed "way off".

The article is factually and technically correct, while your "Because Google probably considers" is speculation.

The "special" feed supported crawling efficiency, giving Google a real time firehouse of new tweets, instead of having to crawl Twitter as any visitor or spider would.

The "no follow" breaks the association between a tweeter and the shared content, exactly as the linked article states.

In Google's own words, "Essentially, using nofollow causes us to drop the target links from our overall graph of the web."

Re: How rel=nofollow Works

#12

Post by Terretta »

Loic wrote: post #8
chalst wrote: post #5

Earlier quoted context omitted.

Google has said that it won't add a URL to its index of URLs to crawl as a result of seeing the URL in a rel=nofollow href [1]. This is over and above not weighting the link if the URL does otherwise make it onto its to crawl list. [1] http://googleblog.blogspot.com/2007/02/robots-exclusion-prot...

Sure, but the probability of having an URL shared only through a tweet is zero. Google is insanely efficient at finding new URLs. So, it will find it anyway, but it will not be shown to you as "shared by your friend John" because of the nofollow.

> the probability of having an URL shared only through a tweet is zero

Countless URLs are shared only through tweets, for example URLs that are private URLs to content on services such as TwitPic[1].

1. http://www.dailymail.co.uk/tvshowbiz/article-2084361/AnnaLyn... (NSFW)

Re: How rel=nofollow Works

#13

Post by luigi »

Loic wrote: post #4

The article is a bit wrong. nofollow does not mean that the crawlers must not follow the link, but means that the ranking algorithms must not consider the link for the ranking as the link could be dubious or spam. As such, the relationship between the page linked and the page linking is not to be affected by the link. And this is what makes the links not surfacing. Basically, the current Twitter HTML says: We have no…

It's true that there's a difference between crawling, indexing, and ranking, but Google is pretty clear on the ultimate effect.

http://support.google.com/webmasters/bin/answer.py?hl=en&#38...

How does Google handle nofollowed links?

In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from our overall graph of the web. However, the target pages may still appear in our index if other sites link to them without using nofollow, or if the URLs are submitted to Google in a Sitemap.

Re: How rel=nofollow Works

#14

Post by masklinn »

chalst wrote: post #3

The cited Danny Sullivan article has a little of his interview with Eric Schmidt: sullivan> I countered that Google seemed to have all the permission it needed, in that they’re not blocked from crawling pages. schmidt>“That’s your opinion,” Schmidt said, then joked: “If you could arrange a letter from Facebook and Twitter to us, that would be helpful.” sullivan> I pushed back that both have effectively given those le…

> The answer for both rel=nofollow and robots.txt is that Google has explicitly promised webmasters that it will not do this.

No, not for a/@rel="nofollow", the only promise Google does is that it will not transfer pagerank credit to the link's destination[0].

This does not prevent them from using that link as an outbound in order to discover new content, nor does it prevent Google from reversing the link for the "Shared On" feature. Google specifically advocates[0] using a/@rel=nofollow for user-generated links in order to prevent rewarding spammers, Twitter's use of the attribute is not just sensible, it's necessary.

niyazpk's comment[1] looks far more sensible: "Shared On" is a specific feature for content producers and part of a special agreement and API access.

[0] http://googleblog.blogspot.com/2005/01/preventing-comment-sp...

[1] http://news.ycombinator.com/item?id=3456233

Re: How rel=nofollow Works

#15

Post by gojomo »

chalst wrote: post #5
Loic wrote: post #4

The article is a bit wrong. nofollow does not mean that the crawlers must not follow the link, but means that the ranking algorithms must not consider the link for the ranking as the link could be dubious or spam. As such, the relationship between the page linked and the page linking is not to be affected by the link. And this is what makes the links not surfacing. Basically, the current Twitter HTML says: We have no…

Google has said that it won't add a URL to its index of URLs to crawl as a result of seeing the URL in a rel=nofollow href [1]. This is over and above not weighting the link if the URL does otherwise make it onto its to crawl list. [1] http://googleblog.blogspot.com/2007/02/robots-exclusion-prot...

Note when Google first unveiled their 'rel=nofollow' invention in 2005, they were very careful to say only that it meant 'no weighting', but not necessarily 'no discovery via this reference'. The attempt to formalize the attribute maintained that same distinction. [1]

The Google manager blogging about it two years later (five years ago) was probably correct about then-current practice, but I wouldn't count on that as an enduring commitment that Google had never and would never follow such links.

[1] http://microformats.org/wiki/rel-nofollow

Re: How rel=nofollow Works

#16

Post by masklinn »

lambada wrote: post #2

I'm amazed really by how much FUD has be4en spread around by Twitter, and most reports of the story. This article clears it up perfectly.

It does not, a/@rel=nofollow has nothing to do with it, the only thing Google says it will do with these is break searchrank credit transference, and Google themselves recommends to set this attribute:

> anywhere that users can add links by themselves, including within comments, trackbacks, and referrer lists

http://googleblog.blogspot.com/2005/01/preventing-comment-sp...

In fact, Google created this attribute specifically and solely to combat comment/user spam.

If Google decided to expand the role of @rel="nofollow" as a gigantic "fuck you" to everybody that's a different issue, but the article still is not right.

Re: How rel=nofollow Works

#17

Post by masklinn »

luigi wrote: post #13
Loic wrote: post #4

The article is a bit wrong. nofollow does not mean that the crawlers must not follow the link, but means that the ranking algorithms must not consider the link for the ranking as the link could be dubious or spam. As such, the relationship between the page linked and the page linking is not to be affected by the link. And this is what makes the links not surfacing. Basically, the current Twitter HTML says: We have no…

It's true that there's a difference between crawling, indexing, and ranking, but Google is pretty clear on the ultimate effect. http://support.google.com/webmasters/bin/answer.py?hl=en&#38... How does Google handle nofollowed links? In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from…

That's the robots nofollow, we're talking about a/@rel=nofollow which google advertises quite differently: http://googleblog.blogspot.com/2005/01/preventing-comment-sp... and the microformats standardization[0] stuck very carefully to "provide no additional weight or ranking to target" period. Not that crawlers can't follow that link for new content, and not that crawlers can't reverse that link for "shared on" features, just that this precise link won't add to the pagerank of the target.

[0] http://microformats.org/wiki/rel-nofollow

Re: How rel=nofollow Works

#18

Post by jasonlotito »

masklinn wrote: post #14
chalst wrote: post #3

The cited Danny Sullivan article has a little of his interview with Eric Schmidt: sullivan> I countered that Google seemed to have all the permission it needed, in that they’re not blocked from crawling pages. schmidt>“That’s your opinion,” Schmidt said, then joked: “If you could arrange a letter from Facebook and Twitter to us, that would be helpful.” sullivan> I pushed back that both have effectively given those le…

> The answer for both rel=nofollow and robots.txt is that Google has explicitly promised webmasters that it will not do this. No, not for a/@rel="nofollow", the only promise Google does is that it will not transfer pagerank credit to the link's destination[0]. This does not prevent them from using that link as an outbound in order to discover new content, nor does it prevent Google from reversing the link for the "Sh…

> the only promise Google does is that it will not transfer pagerank credit to the link's destination

Google is fairly clear about this:

http://support.google.com/webmasters/bin/answer.py?hl=en&#38...

"How does Google handle nofollowed links?

In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from our overall graph of the web. However, the target pages may still appear in our index if other sites link to them without using nofollow, or if the URLs are submitted to Google in a Sitemap. Also, it's important to note that other search engines may handle nofollow in slightly different ways."

So, if your tweet includes a link, they drop that link from the post as far as their graph is concerned. And that graph is what makes up their search.

It's not just page rank.

Re: How rel=nofollow Works

#19

Post by jasonlotito »

masklinn wrote: post #16
lambada wrote: post #2

I'm amazed really by how much FUD has be4en spread around by Twitter, and most reports of the story. This article clears it up perfectly.

It does not, a/@rel=nofollow has nothing to do with it, the only thing Google says it will do with these is break searchrank credit transference, and Google themselves recommends to set this attribute: > anywhere that users can add links by themselves, including within comments, trackbacks, and referrer lists http://googleblog.blogspot.com/2005/01/preventing-comment-sp... In fact, Google created this attribute specif…

You are repeating this misinformation. As I explained here: http://news.ycombinator.com/item?id=3456404 Google makes this clear in this answer: http://support.google.com/webmasters/bin/answer.py?hl=en&#38...

Re: How rel=nofollow Works

#20

Post by jasonlotito »

masklinn wrote: post #17
luigi wrote: post #13

Earlier quoted context omitted.

It's true that there's a difference between crawling, indexing, and ranking, but Google is pretty clear on the ultimate effect. http://support.google.com/webmasters/bin/answer.py?hl=en&#38... How does Google handle nofollowed links? In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from…

That's the robots nofollow, we're talking about a/@rel=nofollow which google advertises quite differently: http://googleblog.blogspot.com/2005/01/preventing-comment-sp... and the microformats standardization[0] stuck very carefully to "provide no additional weight or ranking to target" period . Not that crawlers can't follow that link for new content, and not that crawlers can't reverse that link for "shared on" feat…

Dear god, your basically repeating one link from 7 years ago.

Provided here as information for people who might think you are offering a valid resource:

http://support.google.com/webmasters/bin/answer.py?hl=en&#38...

Post reply on HN