Mine is more awesome: http://logotype.se/robots.txt
Why evil yandex?
Google's robots.txt
51–60 of 94 posts
Re: Google's robots.txt
#52Mine is more awesome: http://logotype.se/robots.txt
Why you consider Yandex evil?
Re: Google's robots.txt
#53Earlier quoted context omitted.
I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker
That's amazing. Know of anything like it for Nginx?
It's blocking by user agent and source ip. You should be able to port the list easily for Nginx. I'd even say you can write a simple awk script in a few minutes to convert from Apache's format to Nginx's.
Re: Google's robots.txt
#54Mine is more awesome: http://logotype.se/robots.txt
Heh, if only Yandex and Baidu respected robots.txt.
Yandex's documentation about implemented robots.txt features:
http://help.yandex.com/webmaster/controlling-robot/robots-tx...
Yandex's online tool to check specific URL on allowness in robots.txt:
Re: Google's robots.txt
#55facebook's... https://www.facebook.com/robots.txt
Bing is allowed. But it's obvious why.
I frequently wonder - is Facebook allowed to say, "bing, you can crawl us. NewCompetitor, you cannot."
I feel like once a company allows public access by posting stuff on the web, they can specify terms, but not include/exclude groups specifically. (In a legal sense; I understand blocking systems that hammer servers but will respect robots.txt. IME bing is the worst offender -- they hammer my sites, send no traffic, but will stop if I specify in robots.txt.)
Does anyone have an opinion about "once public, I can crawl"?
Re: Google's robots.txt
#56Earlier quoted context omitted.
Heh, if only Yandex and Baidu respected robots.txt.
I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker
>Unless your website is written in Russian or Chinese, you probably don't get any traffic from them. They mostly just waste bandwidth and consume resources.
THIS is evil. You could use this argument for banning any new search engine.
Re: Google's robots.txt
#57Apple allows robots access to everything! http://www.apple.com/robots.txt
If you're big enough, doesn't this just make sense? Why waste time maintaining a robots.txt policy when it must represent a tiny fraction of your traffic, which your servers can surely handle? And the really 'bad' guys are going to ignore it anyway. And if you really care, you'll have some much more sophisticated bandwidth throttling in place.
For the smaller guys, sure it makes sense to have some kind of simple robots.txt policy.
Re: Google's robots.txt
#58Earlier quoted context omitted.
I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker
That's amazing. Know of anything like it for Nginx?
curl -s https://raw.github.com/bluedragonz/bad-bot-blocker/master/.h... | awk -F\" '/SetEnvIfNoCase/ {pattern = pattern $2 "|"} END {print "if ($http_user_agent ~ (" substr(pattern, 0, length(pattern)-1) ")) { return 403; }"}'
Not tested. :)
Re: Google's robots.txt
#59facebook's... https://www.facebook.com/robots.txt
I really can't believe facebook is written in php...
That's because it isn't. Or at least not in the traditional sense. It's not just some old bullshit, scripted in PHP, running on an array of scrappy LAMP boxen.
They run their PHP on HHVM for one:
https://en.m.wikipedia.org/wiki/HipHop_for_PHP
https://github.com/facebook/hhvm
...and yeah, it executes PHP code, for sure. But right there, things are already different, and the reality is that they've written a substantial code base in C/C++.
And, two, I'm sure they retain some serious business proprietary trade secrets about their server infrastructure, meaning that while the web front-end might render out HTML like a souped-up CDN, behind the scenes, there is a shit ton of other stuff going down.
Honestly, I think they just leave the file name extensions in the URL for the sake of nostalgia.
Re: Google's robots.txt
#60Mine is more awesome: http://logotype.se/robots.txt
Why you consider Yandex evil?
I don't know if this is universal, but Yandex was quite bad in the past at stumbling around generated links (E.G. calendars that go to infinity) and it wasn't at all unusual to see 400+ links crawled in a day or so. Baidu was the same, but I think they're more well behaved these days.