Google's robots.txt

Post by judk » Sun, Feb 23, 2014, 4:12 AM UTC

mkonecny wrote: post #10

blossoms wrote: post #8

Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt Is this a common mistake?

It just issues a "HTTP/1.1 302 Moved Temporarily" directed to their homepage. Requesting an invalid file such as "robots.txtsdfa32r523" has the same effect, so they probably don't have a robots file at all.

Post by zeckalpha » Sun, Feb 23, 2014, 4:19 AM UTC

yRetsyM wrote: post #5

I remember when this used to be a source of product leaks.

Post by agwa » Sun, Feb 23, 2014, 4:22 AM UTC

russellbeattie wrote: post #12

logotype wrote: post #9

Mine is more awesome: http://logotype.se/robots.txt

Heh, if only Yandex and Baidu respected robots.txt.

Post by drdeadringer » Sun, Feb 23, 2014, 4:27 AM UTC

logotype wrote: post #9

Mine is more awesome: http://logotype.se/robots.txt

Post by khyh » Sun, Feb 23, 2014, 4:31 AM UTC

ryanpetrich wrote: post #3

See also: http://www.google.com/humans.txt