Facebook's robots.txt

Even Facebook's robots.txt has a hatred for my pseudo-anonymous browser settings. Facebook gives me this (for any page): "Sorry, something went wrong. We're working on getting this fixed as soon as we can."

Re: Facebook's robots.txt

#7

Post by toomuchtodo » Sat, Nov 30, 2013, 12:07 AM UTC

pdfcollect wrote: post #3

Is there a way to replace this robots.txt with a null robots.txt? :)

You just ignore the robots.txt file, crawl slowly, and from distributed virtual machines.

Not that you should do that. Robots.txt is a nicety though, the client doesn't have to respect it, and the server doesn't have to allow your HTTP requests.

Re: Facebook's robots.txt

#8

Post by yalogin » Sat, Nov 30, 2013, 12:10 AM UTC

So what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?

Re: Facebook's robots.txt

#9

Post by startling » Sat, Nov 30, 2013, 12:11 AM UTC

decasteve wrote: post #6

Even Facebook's robots.txt has a hatred for my pseudo-anonymous browser settings. Facebook gives me this (for any page): "Sorry, something went wrong. We're working on getting this fixed as soon as we can."

robots.txt isn't enforced.

Re: Facebook's robots.txt

#10

Post by dblacc » Sat, Nov 30, 2013, 12:22 AM UTC

yalogin wrote: post #8

So what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?

I could be wrong but I believe that the the default is that spiders are blocked and only the "User-Agents" listed are allowed to scrape (but not the disallow pages).