21–26 of 26 posts
Earlier quoted context omitted.
If 90% of the content doesn’t get daily attention then the spider can blow out the caches and cause inordinate costs.
Caches? You could not cache anything and 11 TPS would be just fine to run on a gameboy.
what happens when ai scrappers no longer have info to scrap. funny thing - with wasm, the web won't be scrappable.
At least their bots accurately identify themselves in the User-Agent field even when they're ignoring robots.txt, so serverside blocking is on the table for now at least. Bytedances crawler (Bytespider) is another one which disregards robots.txt but still identifies itself, and you probably should block it because it's very aggressive. It's going to get annoying fast when they inevitably go full blackhat and start ma…
Earlier quoted context omitted.
Caches? You could not cache anything and 11 TPS would be just fine to run on a gameboy.
Not if that content is multiple gigbytes of data per file.
I think they'll survive this massive DDOS of 76MBPS.
Same in the case of ReadTheDocs which was linked above, 10TB of traffic over 6 days, an incredible 19.29MBPS sustained. Has humanity even built technology powerful enough to handle this massive assault?!
At least their bots accurately identify themselves in the User-Agent field even when they're ignoring robots.txt, so serverside blocking is on the table for now at least. Bytedances crawler (Bytespider) is another one which disregards robots.txt but still identifies itself, and you probably should block it because it's very aggressive. It's going to get annoying fast when they inevitably go full blackhat and start ma…
Earlier quoted context omitted.
>That seems overly reductive. I qualified my statement but you've chosen to ignore that. I've been paying attention to the Anthropic bots closely for a (relatively) long time and this mastodon group's problems come as a surprise to me based off that lived experience. I don't doubt the truth of their claims. I looked at https://cdn.fosstodon.org/media_attachments/files/112/877/47... and I see the bandwidth used. But l…
> their one limited experience Their report is just the latest in a string from different entities. It's not just one limited experience.