Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic is scraping websites so fast it's causing problems

pivot-to-ai.com

Post reply on HN
Print view View original

1–10 of 26 posts

Re: Anthropic is scraping websites so fast it's causing problems

#3

Post by superkuh »

I've noticed Anthropic bots in my logs for more than a year now and I welcome them. I'd love for their LLM to be better at what I'm interested in. I run my website off my home connection on a desktop computer and I've never had a problem. I'm not saying my dozens of run-ins with the anthropic bots (there have been 3 variations I've seen so far) are totally representative, but they've been respecting my robots.txt.

They even respect extended robots.txt features like,

    User-agent: *
    Disallow: /library/*.pdf$
I make my websites for other people to see. They are not secrets I hoard who's value goes away when copied. The more copies and derivations the better.

I guess ideas like creative commons and sharing go away when the smell of money enters the water. Better lock all your text behind paywalls so the evil corporations won't get it. Just be aware, for every incorporated entity you block you're blocking just as many humans with false positives, if not more. This anti-"scraping" hysteria is mostly profit motivated.

Re: Anthropic is scraping websites so fast it's causing problems

#5

Post by choppaface »

iLoveOncall wrote: post #4

One million hits in 24 hours is only 11 TPS, if that's causing issues, then Anthropic isn't the problem, your application or hosting is.

If 90% of the content doesn’t get daily attention then the spider can blow out the caches and cause inordinate costs.

Re: Anthropic is scraping websites so fast it's causing problems

#6

Post by jsheard »

At least their bots accurately identify themselves in the User-Agent field even when they're ignoring robots.txt, so serverside blocking is on the table for now at least.

Bytedances crawler (Bytespider) is another one which disregards robots.txt but still identifies itself, and you probably should block it because it's very aggressive.

It's going to get annoying fast when they inevitably go full blackhat and start masquerading as normal browser traffic.

Re: Anthropic is scraping websites so fast it's causing problems

#7

Post by ericholscher »

choppaface wrote: post #5
iLoveOncall wrote: post #4

One million hits in 24 hours is only 11 TPS, if that's causing issues, then Anthropic isn't the problem, your application or hosting is.

If 90% of the content doesn’t get daily attention then the spider can blow out the caches and cause inordinate costs.

Yea, this happened to us, and you can see the cache profile of a crawler here:

https://fosstodon.org/@readthedocs/112877477202118215

A bunch of the traffic hit the origin.

Re: Anthropic is scraping websites so fast it's causing problems

#8

Post by ericholscher »

For those saying "just use a CDN", it's not nearly that simple. Even behind a CDN, the crawlers on our site are hitting large files that aren't frequently accessed. This leads to large cache miss rates:

https://fosstodon.org/@readthedocs/112877477202118215

Re: Anthropic is scraping websites so fast it's causing problems

#9

Post by lolpanda »

Cloudflare has a switch to block all the unknown bots other than the well behaved one. Would this be a simple solution to most of the sites? I wonder if the main concern here is that the sites don't want to waste bandwidth/compute for AI bots or they don't want their content to be used for training.
Post reply on HN