You don't scrape Facebook, Facebook scrapes you!
In the US, you catch a cold.
In Soviet Russia, cold catches you!
Facebook's robots.txt
11–20 of 23 posts
Re: Facebook's robots.txt
#12http://www.google.com/robots.txt
/* would have sufficed
Re: Facebook's robots.txt
#13Even Facebook's robots.txt has a hatred for my pseudo-anonymous browser settings. Facebook gives me this (for any page): "Sorry, something went wrong. We're working on getting this fixed as soon as we can."
robots.txt isn't enforced.
Maybe they should be. Gentleman's agreements do not apply to robots.
Re: Facebook's robots.txt
#14http://disqus.com/humans.txt
Also has a funny error 404 when you remove "s"
Uh oh... Something didn't work. > http://disqus.com/human.txt
Re: Facebook's robots.txt
#15http://www.google.com/robots.txt
/* would have sufficed
Re: Facebook's robots.txt
#16Re: Facebook's robots.txt
#17http://disqus.com/humans.txt
What's "bmw" doing on the top of his head ?
Re: Facebook's robots.txt
#18Earlier quoted context omitted.
robots.txt isn't enforced.
Maybe they should be. Gentleman's agreements do not apply to robots.
And how exactly do you propose verifying that the user agent purporting to be Googlebot or Firefox is actually who they are? They're inherently unenforceable.
robots.txt is basically a list of rules that lay out "This is how we'd like you to crawl us. We might stop serving you if you don't comply", rather than a hard-and-fast set of directives that specify how a webcrawler will be guaranteed to behave.
Re: Facebook's robots.txt
#19What is a User Agent: Yeti?
Re: Facebook's robots.txt
#20What is a User Agent: Yeti?
It's the crawler for Naver, a south Korean search engine.