Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nearly 90% of our AI crawler traffic is from ByteDance

haproxy.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#21

Post by jsheard »

jhpacker wrote: post #16

Cloudflare radar, which presumably a much bigger and better sample, reports Bytespider as the #5 AI Crawler behind FB, Amazon, GPTBot, and Google: https://radar.cloudflare.com/explorer?dataSet=ai.bots And that's not including the most of highest volume spiders overall like Googlebot, Bingbot, Yandex, Ahrefs, etc. Not to say it isn't an issue, but that Forture article they reference is pretty alarmist and thin on deta…

The difference is that, AFAIK, those bigger AI crawlers do respect robots.txt. Google even provides a way to opt-out of AI training without opting-out of search indexing.

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#22

Post by mmastrac »

I found that I was getting random bot attacks on progscrape.com with no identifiable bot signature (ie: a signature matching a valid Chrome Desktop client), but at a rate that was only possible via bot. I ended up having to add token buckets by IP/User Agent to help avoid this deluge of traffic.

Agents that trigger the first level of rate-limiting go through a "tarpit" that holds their connection for a bit before serving it which seems to keep most of the bad actors in check. It's impossible to block them via robots.txt, and I'm trying to avoid using too big of a hammer on my CloudFlare settings.

EDIT: checking the logs, it seems that the only bot getting tarpitted right now is OpenAI, and they _do_ have a GPTBot signature:

    2024-10-31T02:30:23.312139Z  WARN progscrape::web: User hit soft rate limit: ratelimit=soft ip="20.171.206.77" browser=Some("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)") method=GET uri=/?search=science.org

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#23

Post by yazzku »

jsheard wrote: post #12
dartos wrote: post #7

Earlier quoted context omitted.

Should we webmasters just start blocking user agents wholesale? I mean except known good actors. I guess known actors would need a verifiable signature

Search engine crawlers do have verifiable signatures, if a client claims to be Googlebot or Bingbot you don't have to take their word for it. https://developers.google.com/search/docs/crawling-indexing/... https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...

But the converse is not true? There is no guarantee the crawler is not amassing data for model training, or that a crawler (AI or otherwise) does not disguise itself as a normal user?

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#24

Post by Narhem »

yazzku wrote: post #3

tl;dr the crawlers do not respect robots.txt or the user agent anymore, but you can drop big bucks on the enterprise HA offering to stop them through other means.

It’s relatively simple to detect crawlers writing one from scratch could take a few weeks if the infrastructure was in place.

With salaries though finding an externally managed solution might be cheaper.

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#25

Post by jsheard »

yazzku wrote: post #23
jsheard wrote: post #12

Earlier quoted context omitted.

Search engine crawlers do have verifiable signatures, if a client claims to be Googlebot or Bingbot you don't have to take their word for it. https://developers.google.com/search/docs/crawling-indexing/... https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...

But the converse is not true? There is no guarantee the crawler is not amassing data for model training, or that a crawler (AI or otherwise) does not disguise itself as a normal user?

Yeah, but traffic appearing to come from normal users can be throttled and/or CAPTCHA'ed while still allowing Google and Bing to crawl to their hearts content so your SEO isn't affected.

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#26

Post by yazzku »

jsheard wrote: post #21
jhpacker wrote: post #16

Cloudflare radar, which presumably a much bigger and better sample, reports Bytespider as the #5 AI Crawler behind FB, Amazon, GPTBot, and Google: https://radar.cloudflare.com/explorer?dataSet=ai.bots And that's not including the most of highest volume spiders overall like Googlebot, Bingbot, Yandex, Ahrefs, etc. Not to say it isn't an issue, but that Forture article they reference is pretty alarmist and thin on deta…

The difference is that, AFAIK, those bigger AI crawlers do respect robots.txt. Google even provides a way to opt-out of AI training without opting-out of search indexing.

And how much do you trust that shit? Has anyone set up a honeypot as an experiment?

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#27

Post by wtf242 »

I had the same issue with TikTok/ByteDance. They were using almost 100gb of my traffic per month.

I now block all ai crawlers at the cloudflare WAF level. On Monday I noticed a HUGE spike in traffic and my site was not handling it well. After a lot of troubleshooting and log parsing, I was getting millions of requests from China that were getting past cloudflare's bot protection.

I ended up having to force a CF managed challenge for the entire country of China to get my site back in a normal working state.

In the past 24 hours CF has blocked 1.66M bot requests. Good luck running a site without using CloudFlare or something similar.

AI crawlers are just out of control

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#28

Post by andrethegiant »

yazzku wrote: post #3

tl;dr the crawlers do not respect robots.txt or the user agent anymore, but you can drop big bucks on the enterprise HA offering to stop them through other means.

[Shameless plug] I'm building a platform[1] that abides by robots.txt, crawl-delay directive, 429s, Retry-After response header, etc out of the box. Polite crawling behavior as a default + centralized caching would decongest the network and be better for website owners.

[1] https://crawlspace.dev

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#29

Post by SoftTalker »

dartos wrote: post #7
yazzku wrote: post #3

tl;dr the crawlers do not respect robots.txt or the user agent anymore, but you can drop big bucks on the enterprise HA offering to stop them through other means.

Should we webmasters just start blocking user agents wholesale? I mean except known good actors. I guess known actors would need a verifiable signature

I would think rate-limiting would be good. Crawlers are not patient enough to operate at the speed of a real human user.

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#30

Post by sghiassy »

manojlds wrote: post #18
sghiassy wrote: post #9

It’s 90% of 1%… title is misleading

No it isn't

It’s in big bold bullets at the top of the article

“Nearly 1% of our total traffic comes from AI crawlers

Close to 90% of that traffic is from Bytespider, by Bytedance (the parent company of TikTok)”

Post reply on HN