> Someone really ought to do something about it.
What is bro proposing here?
LLM scraper bots are overloading acme.com's HTTPS server
11–20 of 66 posts
Re: LLM scraper bots are overloading acme.com's HTTPS server
#12> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…
Is what an offence lol? Bot scraper traffic?
How do you think search engines work?
Re: LLM scraper bots are overloading acme.com's HTTPS server
#13> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…
It's a bit more like a physical business with a "public welcome" policy like a coffee shop going viral and then having tens of thousands of people walking in and taking pictures but not buying coffee. It's disruptive, but not illegal.
Acme.com is welcome to require authentication for all pages but their home page, which would quickly cause the traffic to drop. They don't want to do this - like the coffee shop, they want to be open to public, and for good reasons.
Sometimes the use profile changes dramatically in a short time. 15 years ago, Netflix created the video streaming market and shared bandwidth capacity that had been excessive before wasn't enough. 15 years before that, Google did the same thing when they created search and started driving tremendous traffic to text based websites which had spread through word of mouth before.
Turns out the micro transaction people probably had the right idea.
Re: LLM scraper bots are overloading acme.com's HTTPS server
#14> Nearly all of them were for non-existent pages.
Do any webservers have a feature where they keep a list in memory of files/paths that exist?
Re: LLM scraper bots are overloading acme.com's HTTPS server
#15> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…
adapt or die
waiting on the govt to do something is a path of failure
Re: LLM scraper bots are overloading acme.com's HTTPS server
#16> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…
Because the law deals with intent. The intent for a 12 year old skiddie with a ddos box is to harm someone else's internet. the intent of big scrapers is to collect data. if you want to make the latter illegal then vote for that instead of loading it with the normative baggage of the former.
It's the same problem as why Occupy Wallstreet fell apart: bunch of losers who don't understand the system screech about the system. because they don't understand it, they can't offer any meaningful dialogue about how to fix it beyond screeching.
Re: LLM scraper bots are overloading acme.com's HTTPS server
#17> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…
Normative vs prerogative state [1]. See US v. Swartz compared to Meta use of LibGen for Llama [1] https://en.wikipedia.org/wiki/Dual_state_(model)
So, I knew Aaron and I definitely would not presume to predict what he would have thought, but I’d point out there is a sizeable state space where he should never have been prosecuted, and scraping by others including large commercial companies should not prosecutable on the same grounds.
I repeat what Aaron’s friends and lawyers said at the time: we were going to fight that case, and we were going to win.
Re: LLM scraper bots are overloading acme.com's HTTPS server
#18The only real solution is to put Anubis in front. For me, I just use Cloudflare in front and that suffices. But it's only a few thousand per hour by default. My homeserver can handle that quite well on its own.
Re: LLM scraper bots are overloading acme.com's HTTPS server
#19I suspect part of the issue is that people are still using things like `acme.com` and `demo.com` as an example domain in their documentation and tests instead of relying on `example.com` which is reserved exactly for this purpose [0] [0]: https://www.iana.org/domains/reserved
A small part. On my server AI bots outnumber real visitors 300 to one.
Re: LLM scraper bots are overloading acme.com's HTTPS server
#20> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…
I have added a DB replica server just to keep my website from succumbing to AI bot traffic.