Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's Fascination with My Robots.txt

blog.nytsoi.net

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Facebook's Fascination with My Robots.txt

#31

Post by 13pixels »

Facebook is honestly the least interesting crawler misbehaving right now. The real shift is GPTBot, ClaudeBot, PerplexityBot and a dozen other AI crawlers that don't even identify themselves half the time.

I've been monitoring server logs across ~150 sites and the pattern is striking: AI crawler traffic increased roughly 8x in the last 12 months, but most site owners have no idea because it doesn't show up in analytics. The bots read everything, respect robots.txt maybe 60% of the time, and the content they index directly shapes what ChatGPT or Perplexity recommends to users.

The irony is that robots.txt was designed for a world where crawling meant indexing for search results. Now crawling means training data and real-time retrieval for AI answers. Completely different power dynamic and most robots.txt files haven't adapted.

Re: Facebook's Fascination with My Robots.txt

#33

Post by nazgulsenpai »

Nextgrid wrote: post #5

> Perhaps someone at their end screwed up a loop conditional, but you'd think some monitoring dashboard somewhere would have a warning pop up because of this. If you've been in any big company you'll know things perpetually run in a degraded, somewhat broken mode. They've even made up the term "error budget" because they can't be bothered to fix the broken shit so now there's an acceptable level of brokenness.

In my 3rd year of enterprise now and learned that there are many engineers who will purposefully not fix/improve their problematic applications as a weird sort of job security. It kind of blew up in their faces last year when we moved most of the affected on-premise applications to cloud. Seems like when you introduce tons of friction on-premise it makes the cloud look even better to the suits.

Re: Facebook's Fascination with My Robots.txt

#35

Post by toast0 »

lloydatkinson wrote: post #27

I recently started maintaining a MediaWiki instance for a niche hobbyist community and we'd been struggling with poor server performance. I didn't set the server up, so came into it assuming that the tiny amount of RAM the previous maintainer had given it was the problem. Turns out all of the major AI slop companies had been hounding our wiki constantly for months, and this had resulted in Apache spawning hundreds of…

> this had resulted in Apache spawning hundreds of instances, bringing the whole machine to a halt.

Ugh, such a weird design. At least my experience has been you are better off setting Apache to always run the same number of instances, and tuning that number as appropriate rather than having the instance count fluctuate under load.

Re: Facebook's Fascination with My Robots.txt

#36

Post by lithos »

lloydatkinson wrote: post #27

I recently started maintaining a MediaWiki instance for a niche hobbyist community and we'd been struggling with poor server performance. I didn't set the server up, so came into it assuming that the tiny amount of RAM the previous maintainer had given it was the problem. Turns out all of the major AI slop companies had been hounding our wiki constantly for months, and this had resulted in Apache spawning hundreds of…

Forum/Wiki content probably more likely to be old enough to be from preAI days, meaning they get to avoid the AI inbreeding problem.

Git content likely to have code for the bot to train on.

Re: Facebook's Fascination with My Robots.txt

#37

Post by lloydatkinson »

toast0 wrote: post #35
lloydatkinson wrote: post #27

I recently started maintaining a MediaWiki instance for a niche hobbyist community and we'd been struggling with poor server performance. I didn't set the server up, so came into it assuming that the tiny amount of RAM the previous maintainer had given it was the problem. Turns out all of the major AI slop companies had been hounding our wiki constantly for months, and this had resulted in Apache spawning hundreds of…

> this had resulted in Apache spawning hundreds of instances, bringing the whole machine to a halt. Ugh, such a weird design. At least my experience has been you are better off setting Apache to always run the same number of instances, and tuning that number as appropriate rather than having the instance count fluctuate under load.

Yeah, I did suggest nginx in the past for this project and I think I'll consider it after this saga. For now I've set pretty low limits on Apache.

Re: Facebook's Fascination with My Robots.txt

#39

Post by talkingtab »

1. Put a note in robots.txt that says

"By accessing this file more than one time per second you agree to pay a fee of $0.1 per access plus an additional $0.1 for each previous access each day. This fee will be charged on a per access basis."

2. Run a program that logs the number for Facebook requests and prints a summary and bill.

2. Then get a stamp, envelope and write out a bill for the first day, call it a demand for payment and send it to:

Facebook, Inc. Attn: Security Department/Custodian of Records 1601 S. California Avenue Palo Alto, CA 94304 U.S.A.

You can optionally send this registered mail, where someone has to sign for it.

Corporations such as FaceBook are used to getting their way in court because they can afford lawyers and you cannot. So they have gotten lazy and do not worry about what is fair or legal.

So take them to court when you have a legitimate legal issue. The courts are there to provide redress when you are aggrieved. Right? Use the courts. You can file a small claims action easily. Just make sure you have 1) a legitimate case, 2) evidence 3) have sent them a demand for payment.

Re: Facebook's Fascination with My Robots.txt

#40

Post by RobotToaster »

antonyh wrote: post #11
Ndymium wrote: post #2

For some reason, Facebook has been requesting my Forgejo instance's robots.txt in a loop for the past few days, currently at a speed of 7700 requests per hour. The resource usage is negligible, but I'm wondering why it's happening in the first place and how many other robot files they're also requesting repeatedly. Perhaps someone at Meta broke a loop condition.

As facebookexternalhit is listed in the robots.txt, it does look like it's optimistically rechecking in the hope it's no longer disallowed. That rate of request is obscene though, and falls firmly into the category of Bad Bot.

My guess is it's dutifully obeying it, not storing anything from the site and then exiting, without clearing the site from the crawl queue.
Post reply on HN