Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic is scraping websites so fast it's causing problems

pivot-to-ai.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Anthropic is scraping websites so fast it's causing problems

#11

Post by JohnFen »

superkuh wrote: post #3

I've noticed Anthropic bots in my logs for more than a year now and I welcome them. I'd love for their LLM to be better at what I'm interested in. I run my website off my home connection on a desktop computer and I've never had a problem. I'm not saying my dozens of run-ins with the anthropic bots (there have been 3 variations I've seen so far) are totally representative, but they've been respecting my robots.txt. Th…

> This anti-"scraping" hysteria is mostly profit motivated.

That seems overly reductive.

First, it sounds like you're insinuating that the people claiming the bots are causing actual disruption to their operations are lying. If that's your intent, some amount of evidence for that would be welcome.

Second, lots of people don't want their content to be used to train these models for reasons that have nothing whatsoever to do with money. Trying to avoid contributing to the training of these models is not the equivalent of rejecting the idea of the free exchange of information.

Re: Anthropic is scraping websites so fast it's causing problems

#12

Post by sitkack »

ericholscher wrote: post #7
choppaface wrote: post #5

Earlier quoted context omitted.

If 90% of the content doesn’t get daily attention then the spider can blow out the caches and cause inordinate costs.

Yea, this happened to us, and you can see the cache profile of a crawler here: https://fosstodon.org/@readthedocs/112877477202118215 A bunch of the traffic hit the origin.

Not trying to splain you, but if you can turn off caching for one hit wonders, everything would be greatly improved.

Re: Anthropic is scraping websites so fast it's causing problems

#15

Post by l1n »

> Sites use robots.txt to tell well-behaved web crawlers what data is up for grabs and what data is off limits. Anthropic ignores it and takes your data anyway. That’s even if you’ve updated your robots.txt with the latest configuration details for Anthropic. [404 Media]

doesn't seem supported by the citation, https://www.404media.co/websites-are-blocking-the-wrong-ai-s...

Re: Anthropic is scraping websites so fast it's causing problems

#16

Post by superkuh »

JohnFen wrote: post #11
superkuh wrote: post #3

I've noticed Anthropic bots in my logs for more than a year now and I welcome them. I'd love for their LLM to be better at what I'm interested in. I run my website off my home connection on a desktop computer and I've never had a problem. I'm not saying my dozens of run-ins with the anthropic bots (there have been 3 variations I've seen so far) are totally representative, but they've been respecting my robots.txt. Th…

> This anti-"scraping" hysteria is mostly profit motivated. That seems overly reductive. First, it sounds like you're insinuating that the people claiming the bots are causing actual disruption to their operations are lying. If that's your intent, some amount of evidence for that would be welcome. Second, lots of people don't want their content to be used to train these models for reasons that have nothing whatsoever…

>That seems overly reductive.

I qualified my statement but you've chosen to ignore that. I've been paying attention to the Anthropic bots closely for a (relatively) long time and this mastodon group's problems come as a surprise to me based off that lived experience. I don't doubt the truth of their claims. I looked at https://cdn.fosstodon.org/media_attachments/files/112/877/47... and I see the bandwidth used. But like I said,

>I'm not saying my dozens of run-ins with the anthropic bots (there have been 3 variations I've seen so far) are totally representative,

My take here is that their one limited experience also isn't representative and others are projecting it on to the entire project due to a shifting cultural perception that "scraping" is something weird and bad to be stopped. But it's not. If it were me I'd be checking my webserver config to be sure robots.txt is actually being violated. And I'd check my set per user-agent bandwidth limits in nginx to make sure they matched. That'd solve it. I'm sure the mastodon software has better solutions even if they haven't solved their own DDoS generating problem since 2017 (ref: https://github.com/mastodon/mastodon/issues/4486)

Re: Anthropic is scraping websites so fast it's causing problems

#19

Post by JohnFen »

superkuh wrote: post #16
JohnFen wrote: post #11

Earlier quoted context omitted.

> This anti-"scraping" hysteria is mostly profit motivated. That seems overly reductive. First, it sounds like you're insinuating that the people claiming the bots are causing actual disruption to their operations are lying. If that's your intent, some amount of evidence for that would be welcome. Second, lots of people don't want their content to be used to train these models for reasons that have nothing whatsoever…

>That seems overly reductive. I qualified my statement but you've chosen to ignore that. I've been paying attention to the Anthropic bots closely for a (relatively) long time and this mastodon group's problems come as a surprise to me based off that lived experience. I don't doubt the truth of their claims. I looked at https://cdn.fosstodon.org/media_attachments/files/112/877/47... and I see the bandwidth used. But l…

> their one limited experience

Their report is just the latest in a string from different entities. It's not just one limited experience.

Re: Anthropic is scraping websites so fast it's causing problems

#20

Post by iLoveOncall »

choppaface wrote: post #5
iLoveOncall wrote: post #4

One million hits in 24 hours is only 11 TPS, if that's causing issues, then Anthropic isn't the problem, your application or hosting is.

If 90% of the content doesn’t get daily attention then the spider can blow out the caches and cause inordinate costs.

Caches? You could not cache anything and 11 TPS would be just fine to run on a gameboy.
Post reply on HN