Robots.txt Disallow: 20 Years of Mistakes To Avoid

#1

Post by hornokplease » Mon, Jun 30, 2014, 5:03 PM UTC

Robots.txt Disallow: 20 Years of Mistakes To Avoid
beussery.com

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#2

Post by spaulo12 » Mon, Jun 30, 2014, 5:51 PM UTC

In the past I've created an empty robots.txt just to keep the 404 errors out of my logs...

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#3

Post by sp332 » Mon, Jun 30, 2014, 6:12 PM UTC

Why does Google ignore the crawl delay?

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#4

Post by freddielarge » Mon, Jun 30, 2014, 6:21 PM UTC

fun fact: robots.txt can also be used by attackers to find admin interfaces or other sensitive tidbits that you don't want search engines to crawl

lots of target-detection crawlers will look at robots.txt as the first thing they do to see if there's any fun pages you don't want the other crawlers to see

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#5

Post by pipihu » Mon, Jun 30, 2014, 6:35 PM UTC

The main use for robots.txt is to prevent crawling of infinite URL spaces: http://googlewebmastercentral.blogspot.com.br/2008/08/to-inf...

Alongside tagging links to such resources with nofollow.

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#6

Post by dredge » Mon, Jun 30, 2014, 6:57 PM UTC

The article contains some good observations, but I'm struggling to understand this one:

"Some sites try to communicate with Google through comments in robots.txt"

In the examples given, none appear to be trying to "communicate with Google through comments" - how is including...

  # What's all this then? 
  #   \
  # 
  #    -----
  #   | . . |
  #    -----
  #  \--|-|--/
  #     | |
  #  |-------|

...a "mistake" to avoid? There's no harm in it at all.

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#7

Post by SoftwareMaven » Mon, Jun 30, 2014, 7:07 PM UTC

dredge wrote: post #6

The article contains some good observations, but I'm struggling to understand this one: "Some sites try to communicate with Google through comments in robots.txt" In the examples given, none appear to be trying to "communicate with Google through comments" - how is including... # What's all this then? # \ # # ----- # | . . | # ----- # \--|-|--/ # | | # |-------| ...a "mistake" to avoid? There's no harm in it at all.

I don't think those examples were of people trying to communicate with a crawler. I think they were examples of comments that the owners knew would be thrown away by crawlers.

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#8

Post by Asparagirl » Mon, Jun 30, 2014, 7:09 PM UTC

This article forgot the very worst use of robots.txt:

  User-agent: ia_archiver
  Disallow: /

Those two lines mean that all content hosted on the entire site will be blocked from the Internet Archive (archive.org) WayBack Machine, and the public will be unable to look at any previous versions of the website's content. It wipes out a public view of the past.

Yeah, I'm looking at you, Washington Post: http://www.washingtonpost.com/robots.txt

Banning access to history like that is shameful.

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#9

Post by Istof » Mon, Jun 30, 2014, 7:14 PM UTC

dredge wrote: post #6

The article contains some good observations, but I'm struggling to understand this one: "Some sites try to communicate with Google through comments in robots.txt" In the examples given, none appear to be trying to "communicate with Google through comments" - how is including... # What's all this then? # \ # # ----- # | . . | # ----- # \--|-|--/ # | | # |-------| ...a "mistake" to avoid? There's no harm in it at all.

"Some sites try to communicate with Google through comments in robots.txt"

I thought that was the whole point of robots.txt

Re: Robots.txt Disallow: 20 Years of Mistakes To Avoid

#10

Post by Istof » Mon, Jun 30, 2014, 7:16 PM UTC

500kb limit? you call that short and sweet?