Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM scraper bots are overloading acme.com's HTTPS server

acme.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: LLM scraper bots are overloading acme.com's HTTPS server

#31

Post by kjok »

spiderfarmer wrote: post #19
davidsojevic wrote: post #2

I suspect part of the issue is that people are still using things like `acme.com` and `demo.com` as an example domain in their documentation and tests instead of relying on `example.com` which is reserved exactly for this purpose [0] [0]: https://www.iana.org/domains/reserved

A small part. On my server AI bots outnumber real visitors 300 to one.

How are you measuring this? Does your solution rely on user agent or device fingerprinting? Curious to know what tools are available today and how accurate they are.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#32

Post by [deleted] »

davidsojevic wrote: post #2

I suspect part of the issue is that people are still using things like `acme.com` and `demo.com` as an example domain in their documentation and tests instead of relying on `example.com` which is reserved exactly for this purpose [0] [0]: https://www.iana.org/domains/reserved

[deleted]

Re: LLM scraper bots are overloading acme.com's HTTPS server

#35

Post by mrweasel »

dylan604 wrote: post #26
spiderfarmer wrote: post #22

Earlier quoted context omitted.

I have 6M pages across 8 domains. I have 10 unique IP residential bots per second working hard to scrape every single page.

and how often are those 6M pages changing? how often are those bots finding anything new? why are the bot makers not noticing no difference and just slowing the request down for essentially stale content to them

In March 2025, Drew DeVault wrote a blog post called "Please stop externalizing your costs directly into my face"[1]. I think that is a pretty good guess as to why these bots do not care about frequency of changes, it costs to much.

Every run is basically a fresh run, no state stored, every page is just feed into the machine a new. At least that's my theory.

The AI companies need a full copy of your page, every time they retrain a model. Now they could store that in their own datacenters, but that's a full copy of the internet, in a market where storage costs are already pretty high. So instead, they just externalize the storage cost. If you run a website, a public Gitlab instance, Forgejo, a wiki, a forum, whatever, you basically functions as free offsite storage for the AI companies.

1) https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...

Re: LLM scraper bots are overloading acme.com's HTTPS server

#36

Post by lovehashbrowns »

dylan604 wrote: post #26
spiderfarmer wrote: post #22

Earlier quoted context omitted.

I have 6M pages across 8 domains. I have 10 unique IP residential bots per second working hard to scrape every single page.

and how often are those 6M pages changing? how often are those bots finding anything new? why are the bot makers not noticing no difference and just slowing the request down for essentially stale content to them

On the platform at my work they scrape the same page multiple times, over and over. They do not care to cache anything. And it’s ridiculous to account for because for example for our properties, everything is news-based so warming the cache was as simple as loading the first X articles to get them into cache. But with AI that is not viable because they scrape as much as possible, articles from 2018, 2017. Management doesn’t want to block them though. It’s just suffering through the endless barrage. I was able to do a lot for this like heavier caching even with pgpool but it’s so crazy that this small subset of bots effectively accounts for like 60%+ of our spend.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#37

Post by danaris »

Frieren wrote: post #3

> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…

> Is there no rule of law anymore?

Have you not been paying attention to the news for the past few years?

No, there isn't. If there were, Trump would be in prison, not the Oval Office. And he and the Republican Party have deliberately fostered this environment of corruption and rule-by-wealth so that they can gain more power and even more wealth.

And now they are also backing the AI zealots, and techbros more generally, to ensure that they can do whatever the hell they want, damn the consequences to the rest of the world.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#38

Post by mrweasel »

avazhi wrote: post #12
Frieren wrote: post #3

> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…

Is what an offence lol? Bot scraper traffic? How do you think search engines work?

Search engines appear to care more about being good "Netizens". It's not like GoogleBot never crashed a site, but it's rare. Search engine bots check if they need to back off for a bit, they check etags, notices if page changes infrequently and slow down their crawler frequency.

If you train an LLM, it's not like you keep a copy of every page around, so there's no point to check if you need to re-scrape a page, you do, because you store nothing.

Personally I think people would be pretty indifferent to the new generation of scrapers, AI or other types, if they at least behaved and slowed down if they notice a site struggling. If they had the slightest bit of respect for others on the web, this wouldn't be an issue.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#40

Post by danaris »

davidsojevic wrote: post #24
spiderfarmer wrote: post #19

Earlier quoted context omitted.

A small part. On my server AI bots outnumber real visitors 300 to one.

I don't mean that users are following the links to `acme.com` and `demo.com` type domains in documentation; I mean that bots are likely finding and following many links to them because of their widespread use in documentation. If you search for `site:github.com "acme.com"` in Google, you'll find numerous instances of the domain being used in contrived links in documentation as an example of how URLs might be structur…

That is very possible.

But it is not necessary to see the results that are being described.

If sites like my tiny little browser game, with roughly 120 weekly unique users, are getting absolutely hammered by the scraper-bots (it was, last year, until I put the Wiki behind a login wall; now I still get a significant amount of bot traffic, it's just no longer enough to actually crash the game), then sites that people actually know and consider important like acme.com are very likely to be getting massive deluges of traffic purely from first-order hits.

Post reply on HN