What about using zip bombs? https://idiallo.com/blog/zipbomb-protection
"Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more." https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%2...
Messing with scraper bots
31–40 of 91 posts
Re: Messing with scraper bots
#32Re: Messing with scraper bots
#33This is done very efficiently. If you return anything unexpected, they’ll just drop you and move on.
Re: Messing with scraper bots
#34I have just cut out up ranges that can not connect. I am blocking USA, Asia and Middle East to prevent most malicious accesses
Blocking most of the world's population is one way of reducing malicious traffic
Re: Messing with scraper bots
#35If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there…
# Nothing to hack around here, I’m just a teapot:
location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ {
return 418;
}
error_page 418 /418.html;
No hard block, instead reply to bots the funny HTTP 418 code (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...). That makes filtering logs easier.Live example: https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is WordPress login URL, and it’s commonly blindly requested by bots searching for weak WordPress installs.)
Re: Messing with scraper bots
#36If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there…
I do something quite similar with nginx: # Nothing to hack around here, I’m just a teapot: location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { return 418; } error_page 418 /418.html; No hard block, instead reply to bots the funny HTTP 418 code ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... ). That makes filtering logs easier. Live example: https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is Wor…
Re: Messing with scraper bots
#37If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there…
Re: Messing with scraper bots
#38If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there…
I do something quite similar with nginx: # Nothing to hack around here, I’m just a teapot: location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { return 418; } error_page 418 /418.html; No hard block, instead reply to bots the funny HTTP 418 code ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... ). That makes filtering logs easier. Live example: https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is Wor…
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
Re: Messing with scraper bots
#39Re: Messing with scraper bots
#40Earlier quoted context omitted.
The old scrapers indexed your site so you may get traffic. This benefits you. AI scrapers will plagiarise your work and bring you zero traffic.
Ya make sure you hold dear that grain of sand on a beach of pre-training data that is used to slightly adjust some embedding weights