Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's Fascination with My Robots.txt

blog.nytsoi.net

Post reply on HN
Print view View original

11–20 of 55 posts

Re: Facebook's Fascination with My Robots.txt

#11

Post by antonyh »

Ndymium wrote: post #2

For some reason, Facebook has been requesting my Forgejo instance's robots.txt in a loop for the past few days, currently at a speed of 7700 requests per hour. The resource usage is negligible, but I'm wondering why it's happening in the first place and how many other robot files they're also requesting repeatedly. Perhaps someone at Meta broke a loop condition.

As facebookexternalhit is listed in the robots.txt, it does look like it's optimistically rechecking in the hope it's no longer disallowed. That rate of request is obscene though, and falls firmly into the category of Bad Bot.

Re: Facebook's Fascination with My Robots.txt

#12

Post by dormento »

Has anyone done research on the topic of trying to block these bots by claiming to host illegal material or talking about certain topics? I mean having a few entries in your robots like "/kill-president", "/illegal-music-downloads", "/casino-lucky-tiger-777" etc.

Re: Facebook's Fascination with My Robots.txt

#13

Post by mrweasel »

bot403 wrote: post #8
mrweasel wrote: post #6

Earlier quoted context omitted.

Even if they haven't added any cache control headers, what kind a of lazy Meta engineer designed their crawler with to just pull the same URL multiple times a second? Is this where all that hardware for AI projects is going? To data centers that just uncritically hits the same URL over and over without checking if the content of a site or page has chanced since the last visit then and calculate a proper retry interva…

It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.

I sincerely doubt that search engines run their crawlers on a single machine and they got it figured out.

Re: Facebook's Fascination with My Robots.txt

#14

Post by OliverGuy »

bot403 wrote: post #8
mrweasel wrote: post #6

Earlier quoted context omitted.

Even if they haven't added any cache control headers, what kind a of lazy Meta engineer designed their crawler with to just pull the same URL multiple times a second? Is this where all that hardware for AI projects is going? To data centers that just uncritically hits the same URL over and over without checking if the content of a site or page has chanced since the last visit then and calculate a proper retry interva…

It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.

That's still the same crawler system though. And it's lazy engineering to not build in something to track when you last requested a url.

And it's quite a trivial feature at that.

Re: Facebook's Fascination with My Robots.txt

#15

Post by 1e1a »

delecti wrote: post #10
evv wrote: post #9

Have you considered serving a zip bomb to this user agent?

I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prevent with their current robots.txt.

Could allow only the path to the zip bomb for this user agent.

Re: Facebook's Fascination with My Robots.txt

#16

Post by nine_zeros »

Nextgrid wrote: post #5

> Perhaps someone at their end screwed up a loop conditional, but you'd think some monitoring dashboard somewhere would have a warning pop up because of this. If you've been in any big company you'll know things perpetually run in a degraded, somewhat broken mode. They've even made up the term "error budget" because they can't be bothered to fix the broken shit so now there's an acceptable level of brokenness.

[dead]

Re: Facebook's Fascination with My Robots.txt

#17

Post by pousada »

dormento wrote: post #12

Has anyone done research on the topic of trying to block these bots by claiming to host illegal material or talking about certain topics? I mean having a few entries in your robots like "/kill-president", "/illegal-music-downloads", "/casino-lucky-tiger-777" etc.

Yea I can’t see how that could backfire in any way

Re: Facebook's Fascination with My Robots.txt

#18

Post by FartyMcFarter »

1e1a wrote: post #15
delecti wrote: post #10

Earlier quoted context omitted.

I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prevent with their current robots.txt.

Could allow only the path to the zip bomb for this user agent.

That will work once at most and then quickly get fixed.

Re: Facebook's Fascination with My Robots.txt

#20

Post by Ndymium »

tananaev wrote: post #7

Maybe they’re trying to DDoS it, and once an error is returned, they assume that no robots.txt file exists and then crawl everything else on the site?

While 7700 per hour sounds big, pretty much any dinky server can handle it. So I don't think it's a matter of DDoS. At this point it's just... odd behaviour.
Post reply on HN