Live data from Hacker News

Messing with scraper bots

herman.bearblog.dev

31–40 of 91 posts

Re: Messing with scraper bots

#31
post #8
post #6

What about using zip bombs? https://idiallo.com/blog/zipbomb-protection

"Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more." https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%2...

[deleted]

Re: Messing with scraper bots

#32
This is a fundamental misunderstanding of what those bots are requesting. They aren’t parsing those PHP files, they are using their existence for fingerprinting — they are trying to determine the existence of known vulnerabilities. They probably immediately stop reading after receiving a http response code and discard the remainder of the request packets.

Re: Messing with scraper bots

#33
They’re not scraping for php files, they’re probing for known vulns in popular frameworks, and then using them as entry points for pwning.

This is done very efficiently. If you return anything unexpected, they’ll just drop you and move on.

Re: Messing with scraper bots

#34
post #25
post #20

I have just cut out up ranges that can not connect. I am blocking USA, Asia and Middle East to prevent most malicious accesses

Blocking most of the world's population is one way of reducing malicious traffic

Make sure to block your own IP address to minimize the chance of a social engineering attack.

Re: Messing with scraper bots

#35
post #4

If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there…

I do something quite similar with nginx:

  # Nothing to hack around here, I’m just a teapot:
  location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { 
      return 418; 
  }
  error_page 418 /418.html;
No hard block, instead reply to bots the funny HTTP 418 code (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...). That makes filtering logs easier.

Live example: https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is WordPress login URL, and it’s commonly blindly requested by bots searching for weak WordPress installs.)

Re: Messing with scraper bots

#36
post #4

If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there…

I do something quite similar with nginx: # Nothing to hack around here, I’m just a teapot: location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { return 418; } error_page 418 /418.html; No hard block, instead reply to bots the funny HTTP 418 code ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... ). That makes filtering logs easier. Live example: https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is Wor…

nginx also has "return 444", a special code that makes it drop the connection altogether. This is quite useful if you don't even want to waste any bandwidth serving an error page. You have an image on your error page, which some crappy bots will download over and over again.

Re: Messing with scraper bots

#37
post #4

If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there…

[deleted]

Re: Messing with scraper bots

#38
post #4

If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example: RewriteEngine On # Block requests that reference .php anywhere (path, query, or encoded) RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR] RewriteCond %{QUERY_STRING} \.php [NC,OR] RewriteCond %{THE_REQUEST} \.php [NC] RewriteRule .* - [F,L] Notes: there…

I do something quite similar with nginx: # Nothing to hack around here, I’m just a teapot: location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { return 418; } error_page 418 /418.html; No hard block, instead reply to bots the funny HTTP 418 code ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... ). That makes filtering logs easier. Live example: https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is Wor…

418? Nice I'll think about it ;-) I would, in addition, prefer that "402 Payment Required" would be instantiated for scrapers ...

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...

Re: Messing with scraper bots

#39
post #25

Earlier quoted context omitted.

Blocking most of the world's population is one way of reducing malicious traffic

Make sure to block your own IP address to minimize the chance of a social engineering attack.

Include 127.0.0.1 as well just in case they get into the server.

Re: Messing with scraper bots

#40

Earlier quoted context omitted.

The old scrapers indexed your site so you may get traffic. This benefits you. AI scrapers will plagiarise your work and bring you zero traffic.

Ya make sure you hold dear that grain of sand on a beach of pre-training data that is used to slightly adjust some embedding weights

I have no reason to help the richest companies on earth adjust weights at a cost to myself.
Post reply on HN