11–20 of 91 posts
What about using zip bombs? https://idiallo.com/blog/zipbomb-protection
I have a web crawler and I have both scraping byte limit and timeout, so zip bombs dont bother me much.
https://github.com/rumca-js/crawler-buddy
I think garbage blabber would be more effective.
Don’t get me wrong, but what’s the problem with scrapers? People invest in SEO to become more visible, yet at the same time they fight against “scraper bots.” I’ve always thought the whole point of publicly available information is to be visible. If you want to make money, just put it behind a paywall. Isn’t that the idea?
AI scrapers will plagiarise your work and bring you zero traffic.
Don’t get me wrong, but what’s the problem with scrapers? People invest in SEO to become more visible, yet at the same time they fight against “scraper bots.” I’ve always thought the whole point of publicly available information is to be visible. If you want to make money, just put it behind a paywall. Isn’t that the idea?
The line is "I technically and able to do this" and "I am engaging with a system in good faith".
Public parks are just there and I can technically drive up and dump rubbish there and if they didn't want me to they should have installed a gate and sold tickets.
Many scrapers these days are sort of equivalent in that analogy to people starting entire fleets of waste disposal vehicles that all drive to parks to unload, putting strain on park operations and making the parks a less tenable service in general.
Hm.. why not using dumbed down small, self-hosted LLM networks to feet the big scrapers with bullshit? I'd sacrifice two CPU cores for this just to make their life awful.
What about using zip bombs? https://idiallo.com/blog/zipbomb-protection
"Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more." https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%2...
Otherwise you can also chain compression methods like: "Content-Encoding: gzip gzip".
Hm.. why not using dumbed down small, self-hosted LLM networks to feet the big scrapers with bullshit? I'd sacrifice two CPU cores for this just to make their life awful.
I would make a list of words from each word class, and a list of sentence structures where each item is a word class. Pick a pseudo-random sentence; for each word class in the sentence, pick a pseudo-random word; output; repeat. That should be pretty simple and fast.
I'd think the most important thing though is to add delays to serving the requests. The purpose is to slow the scrapers down, not to induce demand on your garbage well.
Don’t get me wrong, but what’s the problem with scrapers? People invest in SEO to become more visible, yet at the same time they fight against “scraper bots.” I’ve always thought the whole point of publicly available information is to be visible. If you want to make money, just put it behind a paywall. Isn’t that the idea?
The old scrapers indexed your site so you may get traffic. This benefits you. AI scrapers will plagiarise your work and bring you zero traffic.
Don’t get me wrong, but what’s the problem with scrapers? People invest in SEO to become more visible, yet at the same time they fight against “scraper bots.” I’ve always thought the whole point of publicly available information is to be visible. If you want to make money, just put it behind a paywall. Isn’t that the idea?
These scrapers drown peoples' servers in requests, taking up literally all the resources and driving up cost.