Mine is more awesome: http://logotype.se/robots.txt
Along similar lines, personally I was fascinated with https://bridges.torproject.org/robots.txt :)
Google's robots.txt
71–80 of 94 posts
Re: Google's robots.txt
#72Glassdoor uses its for recruiting: http://www.glassdoor.com/robots.txt
Why would they disallow all their about content? (Bit of an SEO noob).
Over-simply, because they do not want to indicate that their site is about those pages, for whatever reason.
In practice, I'm not entirely sure but it looks like it's quite old since those pages don't seem to exist anymore, even as folders which don't exist as pages in themselves, and they aren't 301-redirected to the current relevant pages.
In fact, they're all 404's. So perhaps they used to be pages, were deleted, and kept being crawled which made their site look crap (because of the 404's). Now they could use 301's, but I assume that the reason they didn't is because they might want to restructure the site in the future and re-use those pages. They don't use 302's because 302's are unreliable and freaky.
Does that sound right to everyone else?
Re: Google's robots.txt
#73Mine is more awesome: http://logotype.se/robots.txt
Oh wow, I didn't realise my robots.txt joke had spread!
I added that to the last.fm/robots.txt many years ago (http://www.wired.com/business/2010/08/robot-laws/all/), and have just been made aware that it appears to have spread across the internet:
https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fi...
Favourite sites I've found with it so far: php.net, princeton.edu, nest.com, songkick.com.
Re: Google's robots.txt
#74Re: Google's robots.txt
#75Earlier quoted context omitted.
Bing is allowed. But it's obvious why.
I frequently wonder - is Facebook allowed to say, "bing, you can crawl us. NewCompetitor, you cannot." I feel like once a company allows public access by posting stuff on the web, they can specify terms, but not include/exclude groups specifically. (In a legal sense; I understand blocking systems that hammer servers but will respect robots.txt. IME bing is the worst offender -- they hammer my sites, send no traffic,…
I can think of no reason why there would be any such restriction.
Suppose Facebook is getting paid by Bing, and won't offer crawling to those that aren't paying it? Suppose Facebook considers Baidu's crawler to be evil and chooses to prohibit it for that reason? Suppose Facebook just kind of likes the guys at Bing and decides to allow them special access? If you agree in the first place that Facebook should have the right to put ANY sort of restrictions on who can crawl their side, then why should ANY of these be prohibited? This is not a "common carrier" kind of situation.
Re: Google's robots.txt
#76Earlier quoted context omitted.
Why would they disallow all their about content? (Bit of an SEO noob).
Over-simply, because they do not want to indicate that their site is about those pages, for whatever reason. In practice, I'm not entirely sure but it looks like it's quite old since those pages don't seem to exist anymore, even as folders which don't exist as pages in themselves, and they aren't 301-redirected to the current relevant pages. In fact, they're all 404's. So perhaps they used to be pages, were deleted,…
These guys are the best white-hat SEO growth hackers on the planet, just copy them and don't question it.
Re: Google's robots.txt
#77Mine is more awesome: http://logotype.se/robots.txt
Why you consider Yandex evil?
I IP-banned Chinese search engine crawlers (yandex, baidu etc) because they did not respect the crawl-delay instruction in my robots.txt, causing my low-end crappy php server running vbulletin to bork out. Basically, their bots are too aggressive.
Re: Google's robots.txt
#78Apple allows robots access to everything! http://www.apple.com/robots.txt
If you're big enough, doesn't this just make sense? Why waste time maintaining a robots.txt policy when it must represent a tiny fraction of your traffic, which your servers can surely handle? And the really 'bad' guys are going to ignore it anyway. And if you really care, you'll have some much more sophisticated bandwidth throttling in place. For the smaller guys, sure it makes sense to have some kind of simple robo…
Thats not quite true. Apple has multiple subdomains, each for some part of their site. www.apple.com hosts most of the marketing stuff, but have a look at other pages, e.g.: http://store.apple.com/robots.txt
Re: Google's robots.txt
#79Wikipedia's robots.txt is quite verbose:
http://en.wikipedia.org/robots.txt
Re: Google's robots.txt
#80facebook's... https://www.facebook.com/robots.txt
Bing is allowed. But it's obvious why.
Do you get a different file than me? No-one but ia_archiver is allowed, see https://pastee.org/zpjsa