11–20 of 45 posts
tl;dr the crawlers do not respect robots.txt or the user agent anymore, but you can drop big bucks on the enterprise HA offering to stop them through other means.
Should we webmasters just start blocking user agents wholesale? I mean except known good actors. I guess known actors would need a verifiable signature
https://developers.google.com/search/docs/crawling-indexing/...
https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...
tl;dr the crawlers do not respect robots.txt or the user agent anymore, but you can drop big bucks on the enterprise HA offering to stop them through other means.
Should we webmasters just start blocking user agents wholesale? I mean except known good actors. I guess known actors would need a verifiable signature
It’s 90% of 1%… title is misleading
90% of their crawler traffic (which is 1% of their total traffic) is ByteDance.
Not to say it isn't an issue, but that Forture article they reference is pretty alarmist and thin on detail.
Earlier quoted context omitted.
Their user-agent.
Ah, so this is just marketing.
It is not. We rely on more than User Agents because they are too often faked, so it is not just marketing. There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent.
It’s 90% of 1%… title is misleading
Earlier quoted context omitted.
Ah, so this is just marketing.
(disclaimer: i wrote that post) It is not. We rely on more than User Agents because they are too often faked, so it is not just marketing. There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent.
Great! What are these signals? That seems to be the meat of the post but it's conspicuously absent. How are we supposed to validate the post?
tl;dr the crawlers do not respect robots.txt or the user agent anymore, but you can drop big bucks on the enterprise HA offering to stop them through other means.
Should we webmasters just start blocking user agents wholesale? I mean except known good actors. I guess known actors would need a verifiable signature
"Verifiable signature"? That's a dangerous road to go down, and Google actually wanted to do it (Web Integrity API). Nobody supported them and they backed out.