taobao.com has the shortest robots.txt
http://www.taobao.com/robots.txt
Google's robots.txt
81–90 of 94 posts
Re: Google's robots.txt
#82Curious as to why someone sat down and added this line to that file: Allow: /maps?hq=http://maps.google.com/help/maps/directions/biking/mapleft.kml&ie=UTF8&ll=37.687624,-122.319717&spn=0.346132,0.727158&z=11&lci=bike&dirflg=b&f=d
Here's where that link goes to: http://i.imgur.com/CmdBeQy.png
Re: Google's robots.txt
#83Earlier quoted context omitted.
Over-simply, because they do not want to indicate that their site is about those pages, for whatever reason. In practice, I'm not entirely sure but it looks like it's quite old since those pages don't seem to exist anymore, even as folders which don't exist as pages in themselves, and they aren't 301-redirected to the current relevant pages. In fact, they're all 404's. So perhaps they used to be pages, were deleted,…
These guys are the best white-hat SEO growth hackers on the planet, just copy them and don't question it.
Oh, I wasn't questioning them - just asking if my assessment sounded right.
Re: Google's robots.txt
#84Earlier quoted context omitted.
I really can't believe facebook is written in php...
That's because it isn't. Or at least not in the traditional sense. It's not just some old bullshit, scripted in PHP, running on an array of scrappy LAMP boxen. They run their PHP on HHVM for one: https://en.m.wikipedia.org/wiki/HipHop_for_PHP https://github.com/facebook/hhvm ...and yeah, it executes PHP code, for sure. But right there, things are already different, and the reality is that they've written a substantia…
HipHop is still PHP.
Also their frontend is the only thing written in PHP. You hit the site, you're hitting PHP pages.. Not just an extension for nostalgic reasons.
Re: Google's robots.txt
#85Earlier quoted context omitted.
>Unless your website is written in Russian or Chinese, you probably don't get any traffic from them. They mostly just waste bandwidth and consume resources. THIS is evil. You could use this argument for banning any new search engine.
What "new search engine" has actually generated actual revenue for any webmaster in the past ten years? You could argue DDG but that's the only one I can think of.
I think DDG primarily uses Yandex. They at least often put a "Powered by Yandex" logo along the side.
Re: Google's robots.txt
#86Earlier quoted context omitted.
Why you consider Yandex evil?
I IP-banned Chinese search engine crawlers (yandex, baidu etc) because they did not respect the crawl-delay instruction in my robots.txt, causing my low-end crappy php server running vbulletin to bork out. Basically, their bots are too aggressive.
Yandex is Russian, not Chinese.
Re: Google's robots.txt
#87Curious as to why someone sat down and added this line to that file: Allow: /maps?hq=http://maps.google.com/help/maps/directions/biking/mapleft.kml&ie=UTF8&ll=37.687624,-122.319717&spn=0.346132,0.727158&z=11&lci=bike&dirflg=b&f=d
https://www.google.com/search?num=100&site=&source=hp&q=http...
Turns up three results in google. Very weird indeed.
Re: Google's robots.txt
#88Wikipedia's robots.txt is quite verbose: http://en.wikipedia.org/robots.txt
Because it's part of the Wiki: https://en.wikipedia.org/w/index.php?title=MediaWiki:Robots....
Re: Google's robots.txt
#89Apple allows robots access to everything! http://www.apple.com/robots.txt
If you're big enough, doesn't this just make sense? Why waste time maintaining a robots.txt policy when it must represent a tiny fraction of your traffic, which your servers can surely handle? And the really 'bad' guys are going to ignore it anyway. And if you really care, you'll have some much more sophisticated bandwidth throttling in place. For the smaller guys, sure it makes sense to have some kind of simple robo…
Actually, robot traffic accounts for a much larger share of traffic for any of the big websites than you think.
Re: Google's robots.txt
#90Wikipedia's robots.txt is quite verbose: http://en.wikipedia.org/robots.txt
My favourite part:
> # Folks get annoyed when XfD discussions end up the number 1 google hit for
> # their name.