Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM scraper bots are overloading acme.com's HTTPS server

acme.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: LLM scraper bots are overloading acme.com's HTTPS server

#61

Post by time4tea »

You can block CN, RU, SG, KR, and the level 3 from "ipsum" and the numbers go down a lot.

People might not know about ipset - dont use individual rules in iptables.

Nginx can reject easily based on country.

geoip2 /etc/GeoLite2-Country.mmdb { $geoip2_metadata_country_build metadata build_epoch; $geoip2_data_country_code default=Unknown source=$remote_addr country iso_code; }

  map $geoip2_data_country_code $allowed_country {
    default yes;
    KR no;
    SG no;
    CN no;
    RU no;
 }
server { .... if ($allowed_country = no) { return 444; } }

Re: LLM scraper bots are overloading acme.com's HTTPS server

#62

Post by xena »

janmo wrote: post #50

> I closed port 443 > Now closing https service is obviously just a temporary fix Probably the best starting point would be to edit the robots.txt file and disallow LLM bots there. Currently the file allows all bots: http://acme.com/robots.txt

If the bots respected robots.txt, I would not have a viable business plan.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#63

Post by JohnTHaller »

kjok wrote: post #44
JohnTHaller wrote: post #4

Series of Chinese LLM scrapers kept PortableApps.com running slow and occasionally unresponsive for 2 weeks.

How do you know that they were LLM scrapers? The reason I ask is because user agents could easily be spoofed?

Well known ones. Hundreds of Chinese IPs. Useragents and IPs matched with a year long campaign from them.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#64

Post by danaris »

grumbelbart2 wrote: post #57
danaris wrote: post #40

Earlier quoted context omitted.

That is very possible . But it is not necessary to see the results that are being described. If sites like my tiny little browser game, with roughly 120 weekly unique users, are getting absolutely hammered by the scraper-bots (it was, last year, until I put the Wiki behind a login wall; now I still get a significant amount of bot traffic, it's just no longer enough to actually crash the game), then sites that people…

The article describes that a lot of the requests are for non-existent URLs. Do you observe the same?

Yes; I get a lot of requests for a mostly a small set of paths on my site that look like they're attempts at finding exploitable surfaces. Things like /auth/bind-session, /auth/check?jwt=, etc. (And those are just the ones that are coming up in the obvious error reports; when I go looking at the logs there are more.)

Re: LLM scraper bots are overloading acme.com's HTTPS server

#65

Post by xigoi »

janmo wrote: post #51
Frieren wrote: post #3

> The LLM companies are not picking on me in particular, they are pounding every site on the net. Why is not this a criminal offense? They are hurting business for profit (or for higher valuation as they probably have no profit at all). Why are corporations allowed to do with impunity what could land even a teenager years in prison? Is there no rule of law anymore? The five-year and ten-year penalties kick in only wh…

His robots.txt explicitly allows bots including LLM bots to scrape his site

Permission to scrape is not permission to DDoS.

Re: LLM scraper bots are overloading acme.com's HTTPS server

#66

Post by legohead »

Frieren wrote: post #59
legohead wrote: post #15

Earlier quoted context omitted.

adapt or die waiting on the govt to do something is a path of failure

> waiting on the govt to do something is a path of failure To keep the goverment accountable is a duty of every citizen and the only way to have a functioning society. The failure is to let the goverment be arbitrary and cater to the powerful instead of following the rule of law and applying it equally at all levels.

we have failed a long time ago in that regard
Post reply on HN