31–40 of 45 posts
It’s 90% of 1%… title is misleading
It's completely accurate. 90% of their crawler traffic (which is 1% of their total traffic) is ByteDance.
“”” Nearly 1% of our total traffic comes from AI crawlers
Close to 90% of that traffic is from Bytespider, by Bytedance (the parent company of TikTok) “””
Earlier quoted context omitted.
More than user-agent, because user-agent cannot be trusted.
Great! Well then... how?
As someone who's built multiple (respectful) Web crawlers, for academic research and for respectable commerce, I'm wondering whether abusers are going to make it harder for legitimate crawlers to operate.
Earlier quoted context omitted.
The difference is that, AFAIK, those bigger AI crawlers do respect robots.txt. Google even provides a way to opt-out of AI training without opting-out of search indexing.
And how much do you trust that shit? Has anyone set up a honeypot as an experiment?
full disclosure: worked there [edit: google] a while ago, not in search, not in AI.
Earlier quoted context omitted.
The difference is that, AFAIK, those bigger AI crawlers do respect robots.txt. Google even provides a way to opt-out of AI training without opting-out of search indexing.
And how much do you trust that shit? Has anyone set up a honeypot as an experiment?
Earlier quoted context omitted.
(disclaimer: i wrote that post) It is not. We rely on more than User Agents because they are too often faked, so it is not just marketing. There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent.
> There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent. Great! What are these signals? That seems to be the meat of the post but it's conspicuously absent. How are we supposed to validate the post?
Earlier quoted context omitted.
> There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent. Great! What are these signals? That seems to be the meat of the post but it's conspicuously absent. How are we supposed to validate the post?
I imagine if that information is disclosed, you won’t be able to verify it, as it will be bypassed… because it was disclosed.
I found that I was getting random bot attacks on progscrape.com with no identifiable bot signature (ie: a signature matching a valid Chrome Desktop client), but at a rate that was only possible via bot. I ended up having to add token buckets by IP/User Agent to help avoid this deluge of traffic. Agents that trigger the first level of rate-limiting go through a "tarpit" that holds their connection for a bit before ser…
Earlier quoted context omitted.
(disclaimer: i wrote that post) It is not. We rely on more than User Agents because they are too often faked, so it is not just marketing. There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent.
> There are other signals we see that confirm whether the request came from a "legitimate" AI scraper, or a different scraper with the same user agent. Great! What are these signals? That seems to be the meat of the post but it's conspicuously absent. How are we supposed to validate the post?
Imagine you were a vendor who were trying to trick the author into divulging his methods. Can a stranger on the Internet be trusted?