31–40 of 228 posts
I presume people have logs that indicate the source for them to place blame on AI scrapers. Is anybody making these available for analysis so we can see exactly who is doing this?
Does this author have a big pre-established audience or something? Struggling to understand why this is front-page worthy.
because he's unable to self-host git anymore because AI bots are hammering it to submit PRs. self-hosting was originally a "right" we had upon gaining access to the internet in the 90s, it was the main point of the hyper text transfer protocol.
[flagged]
A varnish cache won't help you if you're running something like a code forge where every commit has its own page - often more than one page, there's the page for the commit and then the page for "history from this commit" and a page for every one of the files that existed in the repo at the time of that commit... Then a poorly written crawler shows up and requests 10,000s of pages that haven't been requested recently…
Just to add further emphasis as to how absurd the current situation is. I host my own repositories with gotd(8) and gotwebd(8) to share within a small circle of people. There is no link on the Internet to the HTTP site served by gotwebd(8), so they fished the subdomain out of the main TLS certificate. I am getting hit once every few seconds for the last six or so months by crawlers ignoring the robots.txt (of course) and wandering aimlessly around "high-value" pages like my OpenBSD repository forks calling blame, diff, etc.
Still managing just fine to serve things to real people, despite me at times having two to three cores running at full load to serve pointless requests. Maybe I will bother to address this at some point as this is melting the ice caps and wearing my disks out, but for now I hope they will choke on the data at some point and that it will make their models worse.
The author of this post could solve their problem with Cloudflare or any of its numerous competitors. Cloudflare will even do it for free.
The scrapers should use some discretion. There are some rather obvious optimizations. Content that is not changing is less likely to change in the future.
At this point, I think we should look at implementing filters that send different response when AI bots are detected or when the clients are abusive. Not just simple response code but one that poisons their training data. Preferably text that elaborates on the anti consumer practices of tech companies. If there is a common text pool used across sites, may be that will get the attention of bot developers and automatic…
Also, spider traps and 42TB zip of death pages work well on poorly written scrapers that ignored robots.txt =3
Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…
I would love to understand this. Just a few years ago badly behaved scrapers were rare enough not to be worth worrying about. Today they are such a menace that hooking any dynamic site up to a pay-to-scale hosting platform like Vercel or Cloud Run can trigger terrifying bills on very short notice. "It's for AI" feels like lazy reasoning for me... but what IS it for? One guess: maybe there's enough of a market now for…
May be everyone is trying to take advantage of the situation before law eventually catches up.
At this point, I think we should look at implementing filters that send different response when AI bots are detected or when the clients are abusive. Not just simple response code but one that poisons their training data. Preferably text that elaborates on the anti consumer practices of tech companies. If there is a common text pool used across sites, may be that will get the attention of bot developers and automatic…
Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…
I don’t think they mean scrapers necessarily driven by LLMs, but scrapers collecting data to train LLMs.