http://www.google.com/baraza/en
What a weird little product. It's like Yahoo Answers, but somehow with even less sorting or categorization.
Google's robots.txt
31–40 of 94 posts
Re: Google's robots.txt
#32http://www.google.com/baraza/en What a weird little product. It's like Yahoo Answers, but somehow with even less sorting or categorization.
http://www.google.com/baraza/en/help?file=inboundsms
They have an SMS companion service that only works in Ghana..
Re: Google's robots.txt
#33Mine is more awesome: http://logotype.se/robots.txt
Heh, if only Yandex and Baidu respected robots.txt.
I'm sure that Baidu respects robots.txt.
This site coolshell.cn is boycotting baidu, so it tells Baiduspider that it doesn't want to be indexed by baidu.
Baidu respects this and doesn't index anything from this website.
Re: Google's robots.txt
#34Checkout youtube.com/robots.txt !
For the lazy: http://www.youtube.com/robots.txt
Re: Google's robots.txt
#35facebook's... https://www.facebook.com/robots.txt
> Notice: Crawling Facebook is prohibited unless you have express written Wow, really? Who put up this sign?
Someone with 80 character lines enforced in their editor? Here's the second line...
> permission. See: http://www.facebook.com/apps/site_scraping_tos_terms.php
Re: Google's robots.txt
#36facebook's... https://www.facebook.com/robots.txt
That really blows my mind. I mean, how can they say that's any kind of "agreement"?
I someone writes a curl/wget script wrapper & points it to the top 10 websites, they don't enter into any kind of written contract or agreement.
Re: Google's robots.txt
#37Checkout youtube.com/robots.txt !
For the lazy: http://www.youtube.com/robots.txt
And for those who don't know the reference: http://www.youtube.com/watch?v=WGoi1MSGu64
Re: Google's robots.txt
#38facebook's... https://www.facebook.com/robots.txt
I really can't believe facebook is written in php...
Re: Google's robots.txt
#39Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt Is this a common mistake?
Yes, from the server side's mishandling of TXT extension. Probably the server put the MIME type in the HTTP header as "HTML" instead of TXT, and the browser renders the page as such.