Google's robots.txt
google.com
Google's robots.txt
1–10 of 94 posts
Re: Google's robots.txt
#2What a weird entry...
https://www.google.com/maps?hq=http://maps.google.com/help/m...
Re: Google's robots.txt
#3See also: http://www.google.com/humans.txt
Re: Google's robots.txt
#4[deleted]
Re: Google's robots.txt
#5I remember when this used to be a source of product leaks.
Re: Google's robots.txt
#6I remember when this used to be a source of product leaks.
Re: Google's robots.txt
#7What a weird entry... https://www.google.com/maps?hq=http://maps.google.com/help/m...
is it a map focused on SF that highlights bikings paths?
Re: Google's robots.txt
#8Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt
Is this a common mistake?
Re: Google's robots.txt
#9Mine is more awesome: http://logotype.se/robots.txt
Re: Google's robots.txt
#10Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt Is this a common mistake?
It just issues a "HTTP/1.1 302 Moved Temporarily" directed to their homepage. Requesting an invalid file such as "robots.txtsdfa32r523" has the same effect, so they probably don't have a robots file at all.