Live data from Hacker News

Facebook's Fascination with My Robots.txt

blog.nytsoi.net

51–55 of 55 posts

Re: Facebook's Fascination with My Robots.txt

#51
post #5

> Perhaps someone at their end screwed up a loop conditional, but you'd think some monitoring dashboard somewhere would have a warning pop up because of this. If you've been in any big company you'll know things perpetually run in a degraded, somewhat broken mode. They've even made up the term "error budget" because they can't be bothered to fix the broken shit so now there's an acceptable level of brokenness.

At Facebook a full outage is accompanied by "first time?" Memes. Unless you are on the specific team responsible you would indeed not really have any reason to care

Re: Facebook's Fascination with My Robots.txt

#52

Is there any downside in just blocking the whole META IP range? I mean they aren't even running a search engine AFAIK. Why would I want them to crawl my website?

The only downside I can think of is that links to your website that are shared on Facebook / WhatsApp / Messenger won't be able to show a rich preview because they won't be able to access the OpenGraph tags on the shared page.

Rich previews are known to cause higher clickthroughs than non-rich previews (if you care about that).

Re: Facebook's Fascination with My Robots.txt

#53
Probably unrelated, but there was an old exploit on facebook using notes where you could paste an image tag with a URL + dynamic parameter an unlimited amount of times and facebook crawlers would hit that link however many times it was pasted in the note, essentially allowing you to DDOS a site with their servers. https://www.radware.com/blog/security/a-flaw-worth-fixing-ho...

Re: Facebook's Fascination with My Robots.txt

#54
post #5

> Perhaps someone at their end screwed up a loop conditional, but you'd think some monitoring dashboard somewhere would have a warning pop up because of this. If you've been in any big company you'll know things perpetually run in a degraded, somewhat broken mode. They've even made up the term "error budget" because they can't be bothered to fix the broken shit so now there's an acceptable level of brokenness.

>they can't be bothered to fix the broken shit Surely it's more likely that it's just cheaper to pay for the errors than to pay to fix the errors. Why fix 10k worth of errors if it'll cost me 100k to fix it?

In my opinion, if something isn’t actually an error, you modify your logging to not log it as an error. Your error logging/alerting pipeline should always stay clean.

If something shows up in there, you should only have 2 options: 1) it’s an actual error and you fix it and make sure it never happens again, or 2) it’s not an error and then you fix it by adjusting the log level to make sure it isn’t one.

If someone suggests an “error budget” on my watch they get the door. You can have a warning budget (and the resources to adjust the log levels or remediation protocols to fix said “errors”) but actual errors should remain errors - otherwise they’re delivering broken software and that’s not what I’m paying them for.

Of course, companies who have the common sense to do this already do it and nobody in their right mind would suggest an “error budget”, but for those that don’t they have a serious problem that needs to be rectified.

The danger otherwise is that you’re making your observability pipeline useless if “errors” no longer actually mean errors. That’s really bad because now it opens the door to actual errors being ignored until it’s too late and then remediation is more costly.

Re: Facebook's Fascination with My Robots.txt

#55

Facebook is honestly the least interesting crawler misbehaving right now. The real shift is GPTBot, ClaudeBot, PerplexityBot and a dozen other AI crawlers that don't even identify themselves half the time. I've been monitoring server logs across ~150 sites and the pattern is striking: AI crawler traffic increased roughly 8x in the last 12 months, but most site owners have no idea because it doesn't show up in analyti…

This matches what I've been noticing. A lot of AI crawler traffic just doesn't show up clearly in typical analytics dashboards, especially when tools aggressively filter or sample.

Part of why I built UXWizz was to avoid black-box filtering and keep control over how traffic is classified. When you own the analytics stack, you get to decide what’s "valid" instead of inheriting someone else's definition.

Post reply on HN