There are only 602 pages of Google.com indexed on Google.com, mostly 'plus' profiles. Quite a few show with this message:
A description for this result is not available because of this site's robots.txt – learn more.
Which is odd.
Google's robots.txt
21–30 of 94 posts
Re: Google's robots.txt
#22There are only 602 pages of Google.com indexed on Google.com, mostly 'plus' profiles. Quite a few show with this message: A description for this result is not available because of this site's robots.txt – learn more. Which is odd.
Google will still index pages blocked by robots.txt, it just won't crawl them (so it can't get a description/preview snippet). It indexes them based on the URL and how people link to them.
Re: Google's robots.txt
#23Checkout youtube.com/robots.txt !
Re: Google's robots.txt
#24Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt Is this a common mistake?
It just issues a "HTTP/1.1 302 Moved Temporarily" directed to their homepage. Requesting an invalid file such as "robots.txtsdfa32r523" has the same effect, so they probably don't have a robots file at all.
Re: Google's robots.txt
#25facebook's... https://www.facebook.com/robots.txt
> Notice: Crawling Facebook is prohibited unless you have express written
Wow, really? Who put up this sign?
Re: Google's robots.txt
#26See also: http://www.google.com/humans.txt
This doesn't work if you are using HTTPS everywhere.
Re: Google's robots.txt
#27facebook's... https://www.facebook.com/robots.txt
That's not even a text file.
Re: Google's robots.txt
#28Yelp's is fun: https://www.yelp.com/robots.txt
Re: Google's robots.txt
#29facebook's... https://www.facebook.com/robots.txt
Bing is allowed. But it's obvious why.
Re: Google's robots.txt
#30Yelp has rules set for the robots: http://www.yelp.com/robots.txt