Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM scraper bots are overloading acme.com's HTTPS server

acme.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: LLM scraper bots are overloading acme.com's HTTPS server

#21

Post by spiderfarmer »

avazhi wrote: post #12
Frieren wrote: post #3

> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…

Is what an offence lol? Bot scraper traffic? How do you think search engines work?

They work because they offer ways to opt out, they honor crawl delay, setting ideal scraping times, IndexNow, etc.

And they give you real, valuable traffic in return.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#22

Post by spiderfarmer »

chupchap wrote: post #10

Bot traffic is crazy even for smaller sites, but still manageable. I was getting 2,000 visitors a day on my infrequently updated website, but after I blocked all the bots via Cloudflare it went back to the normal double digit visitor count.

I have 6M pages across 8 domains. I have 10 unique IP residential bots per second working hard to scrape every single page.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#23

Post by maplethorpe »

Frieren wrote: post #3

> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…

> Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore?

Those laws are intended to protect corporations. If corporations are the ones doing the scraping, it doesn't make sense for the same laws to affect them.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#24

Post by davidsojevic »

spiderfarmer wrote: post #19
davidsojevic wrote: post #2

I suspect part of the issue is that people are still using things like `acme.com` and `demo.com` as an example domain in their documentation and tests instead of relying on `example.com` which is reserved exactly for this purpose [0] [0]: https://www.iana.org/domains/reserved

A small part. On my server AI bots outnumber real visitors 300 to one.

I don't mean that users are following the links to `acme.com` and `demo.com` type domains in documentation; I mean that bots are likely finding and following many links to them because of their widespread use in documentation.

If you search for `site:github.com "acme.com"` in Google, you'll find numerous instances of the domain being used in contrived links in documentation as an example of how URLs might be structured on an arbitrary domain and also in issues to demonstrate a fully qualified URL without giving away the actual domain people were using.

This means that numerous links are pointing to non-existent paths on `acme.com` because of the nature of how people are using them in documentation and examples.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#25

Post by dylan604 »

spiderfarmer wrote: post #19
davidsojevic wrote: post #2

I suspect part of the issue is that people are still using things like `acme.com` and `demo.com` as an example domain in their documentation and tests instead of relying on `example.com` which is reserved exactly for this purpose [0] [0]: https://www.iana.org/domains/reserved

A small part. On my server AI bots outnumber real visitors 300 to one.

That such an absolutely ludicrous thing to hear in a "wtf are these people doing" type of way. I can't imagine a non-social media site would be generating enough traffic to the level that these bots need to be essentially doing continuous scraping. It's just gross to me to be okay with that level of unsophisticated effort that they just do the same thing over and over with zero gain.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#26

Post by dylan604 »

spiderfarmer wrote: post #22
chupchap wrote: post #10

Bot traffic is crazy even for smaller sites, but still manageable. I was getting 2,000 visitors a day on my infrequently updated website, but after I blocked all the bots via Cloudflare it went back to the normal double digit visitor count.

I have 6M pages across 8 domains. I have 10 unique IP residential bots per second working hard to scrape every single page.

and how often are those 6M pages changing? how often are those bots finding anything new? why are the bot makers not noticing no difference and just slowing the request down for essentially stale content to them

Re: LLM scraper bots are overloading acme.com's HTTPS server

#28

Post by devmor »

Frieren wrote: post #3

> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…

Because they have more money.

I've had to deploy a combination of Cloudflare's bot protection and anubis on over 200 domains across 8 different hosting environments in the last 2 months. I have small business clients that couldn't access their sales and support platforms because their websites that normally see tens of thousands of unique sessions per day are suddenly seeing over a million in an hour.

Anthropic and OpenAI were responsible for over 70% of that traffic.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#29

Post by Lerc »

spiderfarmer wrote: post #19
davidsojevic wrote: post #2

I suspect part of the issue is that people are still using things like `acme.com` and `demo.com` as an example domain in their documentation and tests instead of relying on `example.com` which is reserved exactly for this purpose [0] [0]: https://www.iana.org/domains/reserved

A small part. On my server AI bots outnumber real visitors 300 to one.

Where from? And quite frankly why? There are existing training data sets that are large enough for smaller models. Larger models have been focusing on data quality more than quantity. There's limited utility to further indiscriminate widespread scraping,

Re: LLM scraper bots are overloading acme.com's HTTPS server

#30

Post by herecomesthepre »

kristianp wrote: post #27
kristianp wrote: post #14

> Nearly all of them were for non-existent pages. Do any webservers have a feature where they keep a list in memory of files/paths that exist?

Also why are most requests for non existent pages?

Because they are hunting for vulnerable devices and the requests' existence are unique to an application. Like a VoIP appliance for example.

They usually request something deep like /foo/bar/login.html as part of their reconnaissance.

I'm up to 4 pages of filter rules after the massive IP blacklist.

These assholes are also scanning every address on the IPv4 internet and hoovering up the content.

To answer your first question: No, that's the OS's job. But some clever rules could be setup for filtering invalid requests depending on your web server.

Post reply on HN