Robots.txt Disallow: 20 Years of Mistakes To Avoid

The Internet Archive choosing to honor robots.txt is what's 'banning' the access. Both the request not to be crawled and the decision not to crawl are voluntary, but if the Internet Archive decided it wanted to slurp up the Washington Post tomorrow, there's not much they could do to stop it.

My server returns 410 GONE to robots.txt requests.

The robots exclusion protocol is a ridiculous anachronism. I don't use it and neither should you.

yeah, robots.txt is a horrible standard. trust me, i wrote https://www.npmjs.org/package/robotstxt just so that i can really understand what is going on. it's based on https://developers.google.com/webmasters/control-crawl-index...

the article is pretty much correct (although strangely worded at some times), the stuff about "communicating via robotst comments to google" is of course not true. the example he gives are developer jokes, nothing more.

still, you should not use comments in the robots.txt, why?

you can group user agents i.e.:

    User-agent: Googlebot
    User-agent: bingbot
    User-Agent: Yandex
    Disallow: /

Congrats, you have just disallowed googlebot, bingbot and yandox from crawling (not indexing, just crawling)

ok, now:

    User-agent: Googlebot
    #User-agent: bingbot
    User-Agent: Yandex
    Disallow: /

so well, you have definitly blocked yandex, you do not care for bingbot (commented out), but what about googlebot? is googlebot and yandex part of a user-agent group? or is googlebot it's own group and yandex it's own group? if the commented line is interpredted as blank line, then googlebot and yandex are different groups, if it's interpredted are as non existent, they belong together.

they way i read the spec https://developers.google.com/webmasters/control-crawl-index..., this behaviour is undefined. (pleae correct me if i'm wrong)

simple solution: don't use comments in the robots.txt file.

also, please somebody fork and take over https://www.npmjs.org/package/robotstxt it has this undefined behaviour and it also does not follow HTTP 301 requests (which was unspecified when i coded it) and also it tries to do too much (fetching and analysing, it should only do one thing).

by the way, my recommendation is to have a robots.txt file like this

    User-agent: *
    Dissalow: 

    Sitemap: http://www.example.com/your-sitemap-index.xml

and return HTTP 200

why: if you do not have a file there, then at some point in the future suddenly you will return HTTP 500 or HTTP 200 with some response, that can be misleading. also it's quite common that the staging robots.txt file spills over into the real word, this happens as soon as you forget that you have to care about your real robots.txt

also read the spec https://developers.google.com/webmasters/control-crawl-index...

What frustrates me is the number of websites that impose additional restrictions on anything they don't recognize, or worse, websites that impose additional restrictions on (or worse yet, just outright ban) anything that isn't Googlebot.

And people wonder why alternative search engines have such a hard time taking off.

Quora also disallows the ia_archiver agent.

http://www.quora.com/robots.txt

Here is their explanation (in the robots.txt file)

"We opt out of the wayback machine because inclusion would allow people to discover the identity of authors who had written sensitive answers publicly and later had made them anonymous, and because it would prevent authors from being able to remove their content from the internet if they change their mind about publishing it. As far as we can tell, there is no way for sites to selectively programmatically remove content from the archive and so this is the only way for us to protect writers. If they open up an API where we can remove content from the archive when authors remove it from Quora, but leave the rest of the content archived, we would be happy to opt back in. See the page here: https://archive.org/about/exclude.php"

"Meanwhile, if you are looking for an older version of any content on Quora, we have full edit history tracked and accessible in product (with the exception of content that has been removed by the author). You can generally access this by clicking on timestamps, or by appending "/log" to the URL of any content page."