Facebook's Fascination with My Robots.txt
blog.nytsoi.net
Facebook's Fascination with My Robots.txt
1–10 of 55 posts
Re: Facebook's Fascination with My Robots.txt
#2For some reason, Facebook has been requesting my Forgejo instance's robots.txt in a loop for the past few days, currently at a speed of 7700 requests per hour. The resource usage is negligible, but I'm wondering why it's happening in the first place and how many other robot files they're also requesting repeatedly. Perhaps someone at Meta broke a loop condition.
Re: Facebook's Fascination with My Robots.txt
#3Did you try adding a Cache-Control response header?
Re: Facebook's Fascination with My Robots.txt
#4Facebook just decided that instead of loading the robots.txt for every host they intend to crawl, they'll just ignore all the other robots.txt files and then access this one a million times to restore the average.
Re: Facebook's Fascination with My Robots.txt
#5> Perhaps someone at their end screwed up a loop conditional, but you'd think some monitoring dashboard somewhere would have a warning pop up because of this.
If you've been in any big company you'll know things perpetually run in a degraded, somewhat broken mode. They've even made up the term "error budget" because they can't be bothered to fix the broken shit so now there's an acceptable level of brokenness.
Re: Facebook's Fascination with My Robots.txt
#6Did you try adding a Cache-Control response header?
Even if they haven't added any cache control headers, what kind a of lazy Meta engineer designed their crawler with to just pull the same URL multiple times a second?
Is this where all that hardware for AI projects is going? To data centers that just uncritically hits the same URL over and over without checking if the content of a site or page has chanced since the last visit then and calculate a proper retry interval. Search engine crawlers 25 - 30 years ago could do this.
Hit the URL once per day, if it chances daily, try twice a day. If it hasn't chanced in a week, maybe only retry twice per week.
Re: Facebook's Fascination with My Robots.txt
#7Maybe they’re trying to DDoS it, and once an error is returned, they assume that no robots.txt file exists and then crawl everything else on the site?
Re: Facebook's Fascination with My Robots.txt
#8Did you try adding a Cache-Control response header?
Even if they haven't added any cache control headers, what kind a of lazy Meta engineer designed their crawler with to just pull the same URL multiple times a second? Is this where all that hardware for AI projects is going? To data centers that just uncritically hits the same URL over and over without checking if the content of a site or page has chanced since the last visit then and calculate a proper retry interva…
It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.
Re: Facebook's Fascination with My Robots.txt
#9Have you considered serving a zip bomb to this user agent?
Re: Facebook's Fascination with My Robots.txt
#10Have you considered serving a zip bomb to this user agent?
I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prevent with their current robots.txt.