> Perhaps someone at their end screwed up a loop conditional, but you'd think some monitoring dashboard somewhere would have a warning pop up because of this. If you've been in any big company you'll know things perpetually run in a degraded, somewhat broken mode. They've even made up the term "error budget" because they can't be bothered to fix the broken shit so now there's an acceptable level of brokenness.
Facebook's Fascination with My Robots.txt
51–55 of 55 posts
Re: Facebook's Fascination with My Robots.txt
#52Is there any downside in just blocking the whole META IP range? I mean they aren't even running a search engine AFAIK. Why would I want them to crawl my website?
Rich previews are known to cause higher clickthroughs than non-rich previews (if you care about that).
Re: Facebook's Fascination with My Robots.txt
#53Re: Facebook's Fascination with My Robots.txt
#54> Perhaps someone at their end screwed up a loop conditional, but you'd think some monitoring dashboard somewhere would have a warning pop up because of this. If you've been in any big company you'll know things perpetually run in a degraded, somewhat broken mode. They've even made up the term "error budget" because they can't be bothered to fix the broken shit so now there's an acceptable level of brokenness.
>they can't be bothered to fix the broken shit Surely it's more likely that it's just cheaper to pay for the errors than to pay to fix the errors. Why fix 10k worth of errors if it'll cost me 100k to fix it?
If something shows up in there, you should only have 2 options: 1) it’s an actual error and you fix it and make sure it never happens again, or 2) it’s not an error and then you fix it by adjusting the log level to make sure it isn’t one.
If someone suggests an “error budget” on my watch they get the door. You can have a warning budget (and the resources to adjust the log levels or remediation protocols to fix said “errors”) but actual errors should remain errors - otherwise they’re delivering broken software and that’s not what I’m paying them for.
Of course, companies who have the common sense to do this already do it and nobody in their right mind would suggest an “error budget”, but for those that don’t they have a serious problem that needs to be rectified.
The danger otherwise is that you’re making your observability pipeline useless if “errors” no longer actually mean errors. That’s really bad because now it opens the door to actual errors being ignored until it’s too late and then remediation is more costly.
Re: Facebook's Fascination with My Robots.txt
#55Facebook is honestly the least interesting crawler misbehaving right now. The real shift is GPTBot, ClaudeBot, PerplexityBot and a dozen other AI crawlers that don't even identify themselves half the time. I've been monitoring server logs across ~150 sites and the pattern is striking: AI crawler traffic increased roughly 8x in the last 12 months, but most site owners have no idea because it doesn't show up in analyti…
Part of why I built UXWizz was to avoid black-box filtering and keep control over how traffic is classified. When you own the analytics stack, you get to decide what’s "valid" instead of inheriting someone else's definition.