Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM scraper bots are overloading acme.com's HTTPS server

acme.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: LLM scraper bots are overloading acme.com's HTTPS server

#41

Post by MartijnHols »

kjok wrote: post #34

For those who have deployed Cloudflare in front, what are pros and cons? How's the user experience? Do they offer free bot protection?

I opted for Bunny Shield exactly to combat bots, in particular ones that spoof User Agents and rotate millions of IPs. It works great, detecting the vast majority of bots and challenging them. Much more user friendly than Cloudflare too, which typically resorts to challenging everyone (not that CF was ever an option due to various concerns).

I also added various rate limits such as 1 RPS to my expensive SSR pages, after which a visitor gets challenged. Again this blocks bots without harming power users much.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#42

Post by danaris »

Lerc wrote: post #29
spiderfarmer wrote: post #19

Earlier quoted context omitted.

A small part. On my server AI bots outnumber real visitors 300 to one.

Where from? And quite frankly why? There are existing training data sets that are large enough for smaller models. Larger models have been focusing on data quality more than quantity. There's limited utility to further indiscriminate widespread scraping,

Tell that to the idiots doing the scraping.

Small site operators like us know very well that the utility they can get by scraping us is marginal at best. Based on their patterns of behavior, though, my best guess is that they've simply configured their bots to scrape absolutely everything, all the time, forever, as aggressively as possible, and treat any attempt to indicate "hey, this data isn't useful to you" as an adversarial signal that the site operator is trying to hide things from them that are their God-given right.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#43

Post by unethical_ban »

kristianp wrote: post #14

> Nearly all of them were for non-existent pages. Do any webservers have a feature where they keep a list in memory of files/paths that exist?

That's called a WAF, web application firewall, a separate piece of software (or server module) where paths in the web applications hosted are defined, often variables and variable types can be validated, etc. to prevent the kind of attacks these scans are often doing.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#46

Post by spiderfarmer »

lovehashbrowns wrote: post #36
dylan604 wrote: post #26

Earlier quoted context omitted.

and how often are those 6M pages changing? how often are those bots finding anything new? why are the bot makers not noticing no difference and just slowing the request down for essentially stale content to them

On the platform at my work they scrape the same page multiple times, over and over. They do not care to cache anything. And it’s ridiculous to account for because for example for our properties, everything is news-based so warming the cache was as simple as loading the first X articles to get them into cache. But with AI that is not viable because they scrape as much as possible, articles from 2018, 2017. Management…

Many are using residential proxies now. It's impossible to block them. Not even Google Analytics succeeds. People are sitting on reports thinking their website is suddenly very popular, but it's all random ips, from random locations across the world requesting 1 page at a time, at random times of the day.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#47

Post by frankacter »

kjok wrote: post #34

For those who have deployed Cloudflare in front, what are pros and cons? How's the user experience? Do they offer free bot protection?

Some pros for us, in addition to bot protection.

* global distributed caching of content. This reduces the static load on our servers and bandwidth usages to essentially 0, and since it is served at an end point closest to wherever the client is, they get less latency. This includes user logged in specifics as well.

* shared precached common libraries (ie. jquery, etc) for faster client load times

* Offers automated minification of JS, CSS, and HTML, along with image optimization (serve size and resolution of image specific to the device user is viewing it from) to increase speed

* always up mode (even if my server is down for some reason, I can continue to serve static content)

* detailed analytics and reporting on usage / visitors

There are a lot more, but those are a few that come to mind.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#48

Post by qmarchi »

will4274 wrote: post #13
Frieren wrote: post #3

> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…

It's a bit more like a physical business with a "public welcome" policy like a coffee shop going viral and then having tens of thousands of people walking in and taking pictures but not buying coffee. It's disruptive, but not illegal. Acme.com is welcome to require authentication for all pages but their home page, which would quickly cause the traffic to drop. They don't want to do this - like the coffee shop, they w…

Depends on the country. In Japan, you could be considered a "public nusicance" and be tossed behind bars for a bit.
Post reply on HN