Nearly 90% of our AI crawler traffic is from ByteDance
1–10 of 45 posts
Re: Nearly 90% of our AI crawler traffic is from ByteDance
#2How do you differentiate between "ai" (whatever that means) and other crawlers?
Re: Nearly 90% of our AI crawler traffic is from ByteDance
#3tl;dr the crawlers do not respect robots.txt or the user agent anymore, but you can drop big bucks on the enterprise HA offering to stop them through other means.
Re: Nearly 90% of our AI crawler traffic is from ByteDance
#4How do you differentiate between "ai" (whatever that means) and other crawlers?
Their user-agent.
Re: Nearly 90% of our AI crawler traffic is from ByteDance
#5How do you differentiate between "ai" (whatever that means) and other crawlers?
You don't. Theoretically, they would respect the user agent, but who can trust that anymore?
Re: Nearly 90% of our AI crawler traffic is from ByteDance
#6[dead]
Re: Nearly 90% of our AI crawler traffic is from ByteDance
#7tl;dr the crawlers do not respect robots.txt or the user agent anymore, but you can drop big bucks on the enterprise HA offering to stop them through other means.
Should we webmasters just start blocking user agents wholesale?
I mean except known good actors.
I guess known actors would need a verifiable signature
Re: Nearly 90% of our AI crawler traffic is from ByteDance
#8Re: Nearly 90% of our AI crawler traffic is from ByteDance
#9It’s 90% of 1%… title is misleading