Ban me at the IP level if you don't like me

Post by 0x457 » Mon, Aug 25, 2025, 9:00 PM UTC

NegativeK wrote: post #314

kiitos wrote: post #303

Earlier quoted context omitted.

every single IPv4 address in existence receives constant malicious traffic, from uncountably many malicious actors, on all common service ports (80, 443, 22, etc.) and, for HTTP specifically, to an enormous and growing number of common endpoints (mostly WordPress related, last I checked) if you put your server up on the public internet then this is just table stakes stuff that you always need to deal with, doesn't re…

I've heard this point raised elsewhere, and I think it's underplaying the magnitude of the issue. Background scanner noise on the internet is incredibly common, but the AI scraping is not at the same level. Wikipedia has published that their infrastructure costs have notably shot up since LLMs started scraping them. I've seen similar idiotic behavior on a small wiki I run; a single AI company took the data usage from…

Post by ghxst » Mon, Aug 25, 2025, 9:11 PM UTC

NegativeK wrote: post #318

8organicbits wrote: post #235

I've been working on a web crawler and have been trying to make it as friendly as possible. Strictly checking robots.txt, crawling slowly, clear identification in the User Agent string, single IP source address. But I've noticed some anti-bot tricks getting applied to the robot.txt file itself. The latest was a slow loris approach where it takes forever for robots.txt to download. I accidentally treated this as a 404…

I really appreciate you giving a shit. Not sarcastically -- it seems like you're actually doing everything right, and it makes a difference. Gating robots.txt might be a mistake, but it also might be a quick way to deal with crawlers who mine robots.txt for pages that are more interesting. It's also a page that's never visited by humans. So if you make it a tarpit, you both refuse to give the bot more information and…

Post by maxbond » Mon, Aug 25, 2025, 9:22 PM UTC

socksy wrote: post #289

dmesg wrote: post #255

Earlier quoted context omitted.

Does an attacking bot know your webserver is not a misconfigured router exposing its web interface to the net? I often am baffled what conclusions people come up with from half reading posts. I had bots attack me with SSH 2.0 login attempts on port 80 and 443. Some people underestimate how bad at computer science some skids are.

Also baffled that three separate people came to that conclusion. Do they not run web servers on the open web or something? Script kiddies are constantly probing urls, and urls come up in your logs. Sure it would be bad if that was how your app was architected. But it's not how it's architected, it's how the skids hope your app is architected. It's not like if someone sends me a request for /wp-login.php that my rails…

Post by ronsor » Mon, Aug 25, 2025, 9:32 PM UTC

conradludgate wrote: post #387

ronsor wrote: post #370

Earlier quoted context omitted.

> The latest was a slow loris approach where it takes forever for robots.txt to download. I'd treat this in a client the same way as I do in a server application. If the peer is behaving maliciously or improperly, I silently drop the TCP connection without notifying the other party. They can waste their resources by continuing to send bytes for the next few minutes until their own TCP stack realizes what happens.

How do you silently drop a TCP connection? Closing the socket fd usually results in a FIN packet being sent whether I want it to or not. Additionally, it's not going to be using that many resources before your kernel sends it a RST next time a data packet is sent

Post by hinkley » Mon, Aug 25, 2025, 9:59 PM UTC

sidewndr46 wrote: post #324

bob1029 wrote: post #132

I think a lot of really smart people are letting themselves get taken for a ride by the web scraping thing. Unless the bot activity is legitimately hammering your site and causing issues (not saying this isn't happening in some cases), then this mostly amounts to an ideological game of capture the flag. The difference being that you'll never find their flag. The only thing you win by playing is lost time. The best wa…

I don't think you have any idea how serious the issue is. I was loosely speaking in charge of application-level performance at one job for a web app. I was asked to make the backend as fast as possible at dumping the last byte of HTML back to the user. The problem I ran into was performance was bimodal. We had this one group of users that was lightning fast and the rest were far slower. I chased down a few obvious ou…