Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nearly 90% of our AI crawler traffic is from ByteDance

haproxy.com

Post reply on HN
Print view View original

41–45 of 45 posts

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#41

Post by mmastrac »

atif089 wrote: post #39
mmastrac wrote: post #22

I found that I was getting random bot attacks on progscrape.com with no identifiable bot signature (ie: a signature matching a valid Chrome Desktop client), but at a rate that was only possible via bot. I ended up having to add token buckets by IP/User Agent to help avoid this deluge of traffic. Agents that trigger the first level of rate-limiting go through a "tarpit" that holds their connection for a bit before ser…

Did you implement this in your web server or within your application? I'd love to see the code if you're willing to share

Yep, it's open-source:

https://github.com/progscrape/progscrape/blob/master/web/src...

Here's where we handle the rate limits:

https://github.com/progscrape/progscrape/blob/master/web/src...

I actually misremembered my implementation. It's rolling counting bloom filters, not a token bucket. :)

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#42

Post by superkuh »

jcat123 wrote: post #17
PittleyDunkin wrote: post #10

Earlier quoted context omitted.

Ah, so this is just marketing.

(disclaimer: i wrote that post) It is not. We rely on more than User Agents because they are too often faked, so it is not just marketing. There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent.

So user-agent and whois to see if it's coming from a plausible netblock and accept: strings, http header stuff?

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#43

Post by yazzku »

Arnt wrote: post #35
yazzku wrote: post #26

Earlier quoted context omitted.

And how much do you trust that shit? Has anyone set up a honeypot as an experiment?

You can trust Google to do what it says, and yes I've seen Google obey robots.txt. You can't trust Google to do what you think is right.

No, you can't: https://apnews.com/article/828aefab64d4411bac257a07c1af0ecb

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#44

Post by dgfitz »

PittleyDunkin wrote: post #38
dgfitz wrote: post #36

Earlier quoted context omitted.

I imagine if that information is disclosed, you won’t be able to verify it, as it will be bypassed… because it was disclosed.

What a wonderful world we live in where serious people are expected to believe press releases based purely on brand prestige.

There’s a lot of assumptions in that comment.

Re: Nearly 90% of our AI crawler traffic is from ByteDance

#45

Post by Arnt »

yazzku wrote: post #43
Arnt wrote: post #35

Earlier quoted context omitted.

You can trust Google to do what it says, and yes I've seen Google obey robots.txt. You can't trust Google to do what you think is right.

No, you can't: https://apnews.com/article/828aefab64d4411bac257a07c1af0ecb

I'm a bit in a hurry, don't have time for close reading. Does that article say some Google apps (notably Maps) store locations on your device even if you have configured them to not store it in your Google account? I may miss something, don't have time to read between the lines today.
Post reply on HN