Ban me at the IP level if you don't like me • Hacker News Board

Re: Ban me at the IP level if you don't like me

#321

Post by gabeio » Mon, Aug 25, 2025, 4:10 PM UTC

NegativeK wrote: post #318

8organicbits wrote: post #235

I've been working on a web crawler and have been trying to make it as friendly as possible. Strictly checking robots.txt, crawling slowly, clear identification in the User Agent string, single IP source address. But I've noticed some anti-bot tricks getting applied to the robot.txt file itself. The latest was a slow loris approach where it takes forever for robots.txt to download. I accidentally treated this as a 404…

I really appreciate you giving a shit. Not sarcastically -- it seems like you're actually doing everything right, and it makes a difference. Gating robots.txt might be a mistake, but it also might be a quick way to deal with crawlers who mine robots.txt for pages that are more interesting. It's also a page that's never visited by humans. So if you make it a tarpit, you both refuse to give the bot more information and…

> It's also a page that's never visited by humans.

Never is a strong word. I have definitely visited robots.txt of various websites for a variety of random reasons.

  - remembering the format
  - seeing what they might have tried to "hide"
  - using it like a site's directory
  - testing if the website is working if their main dashboard/index is offline

Re: Ban me at the IP level if you don't like me

#322

Post by SoftTalker » Mon, Aug 25, 2025, 4:12 PM UTC

zeta0134 wrote: post #298

wvbdmp wrote: post #148

Earlier quoted context omitted.

You log passwords?

Just about nobody logs passwords on purpose. But really stupid IoT devices accept credentials as like query strings, or part of the path or something, and it's common to log those. The attacker is sending you passwords meant for a much less secure system.

You probably shouldn't log usernames then, or really any form fields, as users might accidentally enter a password into one of them. Kind of defeats the point of web forms, but safety is important!

Re: Ban me at the IP level if you don't like me

#323

Post by johnklos » Mon, Aug 25, 2025, 4:12 PM UTC

This is a problem.

There's a recent phishing campaign with sites hosted by Cloudflare and spam sent through either "noobtech.in" (103.173.40.0/24) or through "worldhost.group" (many, many networks).

"noobtech.in" has no web site, can't accept abuse complaints (their email has spam filters), and they don't respond at all to email asking them for better communication methods. The phishing domains have "mail.(phishing domain)" which resolves back to 103.173.40.0/24. Their upstream is a Russian network that doesn't respond to anything. It's 100% clear that this network is only used for phishing and spam.

It's trivial to block "noobtech.in".

"worldhost.group", though, is a huge hosting conglomerate that owns many, many hosting companies and many, many networks spread across many ASNs. They do not respond to any attempts to communicate with them, but since their web site redirects to "hosting.com", I've sent abuse complaints to them. "hosting.com" has autoresponders saying they'll get back to me, but so far not a single ticket has been answered with anything but the initial autoresponder.

It's really, really difficult to imagine how one would block them, and also difficult to imagine what kind of collateral impact that'd have.

These huge providers, Tencent included, get away with way too much. You can't communicate with them, they don't give the slightest shit about harmful, abusive and/or illegal behavior from their networks, and we have no easy way to simply block them.

I think we, collectively, need to start coming up with things we can do that would make their lives difficult enough for them to take notice. Should we have a public listing of all netblocks that belong to such companies and, as an example, we could choose to autorespond to all email from "worldhost.group" and redirect all web browsing from Tencent so we can tell people that their ISP is malicious?

I don't know what the solution is, but I'd love to feel a bit less like I have no recourse when it comes to these huge mega-corporations.

Re: Ban me at the IP level if you don't like me

#324

Post by sidewndr46 » Mon, Aug 25, 2025, 4:14 PM UTC

bob1029 wrote: post #132

I think a lot of really smart people are letting themselves get taken for a ride by the web scraping thing. Unless the bot activity is legitimately hammering your site and causing issues (not saying this isn't happening in some cases), then this mostly amounts to an ideological game of capture the flag. The difference being that you'll never find their flag. The only thing you win by playing is lost time. The best wa…

I don't think you have any idea how serious the issue is. I was loosely speaking in charge of application-level performance at one job for a web app. I was asked to make the backend as fast as possible at dumping the last byte of HTML back to the user.

The problem I ran into was performance was bimodal. We had this one group of users that was lightning fast and the rest were far slower. I chased down a few obvious outliers (that one forum thread with 11000 replies that some guy leaves up on a browser tab all the time, etc.) but it was still bimodal. Eventually I just changed the application level code to display known bots as one performance trace and everything else as another trace.

60% of all requests are known bots. This doesn't even count the random ass bot that some guy started up at an ISP. Yes, this really happened. We were paying customer of a company who decided to just conduct a DoS attack on us at 2 PM one afternoon. It took down the website.

Not only that, the bots effectively always got a cached response since they all seemed to love to hammer the same pages. Users never got a cached response, since LRU cache eviction meant the actual discussions with real users were always evicted. There were bots that would just rescrape every page they had ever seen every few minutes. There were bots that would just increase their throughput until the backend app would start to slow down.

There were bots that would run the javascript for whatever insane reason and start emulating users submitting forms, etc.

You probably are thinking "but you got to appear in a search index so it is worth it". Not really. Google's bot was one of the few well behaved ones and would even slow scraping if it saw a spike in the response times. Also we had an employee who was responsible for categorizing our organic search performance. While we had a huge amount of traffic from organic search, it was something like 40% to just one URL.

Retrospectively I'm now aware that a bunch of this was early stage AI companies scraping the internet for data.

Re: Ban me at the IP level if you don't like me

#325

Post by SoftTalker » Mon, Aug 25, 2025, 4:15 PM UTC

dmesg wrote: post #255

JohnFen wrote: post #244

Earlier quoted context omitted.

How are passwords ending up in your logs? Something is very, very wrong there.

Does an attacking bot know your webserver is not a misconfigured router exposing its web interface to the net? I often am baffled what conclusions people come up with from half reading posts. I had bots attack me with SSH 2.0 login attempts on port 80 and 443. Some people underestimate how bad at computer science some skids are.

Running ssh on 80 or 443 is a way to get around boneheaded firewalls that allow http(s) but block ssh, so it's not completely insane to see probes for it.

Re: Ban me at the IP level if you don't like me

#326

Post by sidewndr46 » Mon, Aug 25, 2025, 4:16 PM UTC

kiitos wrote: post #303

phito wrote: post #134

Earlier quoted context omitted.

My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it does not impact his service, at the very least it feels like harassment

every single IPv4 address in existence receives constant malicious traffic, from uncountably many malicious actors, on all common service ports (80, 443, 22, etc.) and, for HTTP specifically, to an enormous and growing number of common endpoints (mostly WordPress related, last I checked) if you put your server up on the public internet then this is just table stakes stuff that you always need to deal with, doesn't re…

I was kind of amazed to learn that apparently if you connect Windows NT4/98/2000/ME to a public IPv4 address it gets infected by what is a period correct worm in no time at all. I don't mean that someone uses an RCE to turn it into part of a botnet (that is expected), apparently there are enough infected hosts from 20+ years ago still out there that the sasser worm is still spreading.

Re: Ban me at the IP level if you don't like me

#327

Post by sidewndr46 » Mon, Aug 25, 2025, 4:17 PM UTC

ralferoo wrote: post #265

phito wrote: post #134

Earlier quoted context omitted.

My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it does not impact his service, at the very least it feels like harassment

What's worse is when you get bots blasting HTTP traffic at every open port, even well known services like SMTP. Seriously, it's a mail server. It identified itself as soon as the connection was opened, if they waited 100ms-300ms before spamming, they'd know that it wasn't HTTP because the other side wouldn't send anything at all if it was. There's literally no need to bombard a mail server on a well known port by con…

It's even funnier when you realize it is a request for a known exploit in WordPress. Does someone really run that on port 22?

Re: Ban me at the IP level if you don't like me

#328

Post by sdenton4 » Mon, Aug 25, 2025, 4:25 PM UTC

gabeio wrote: post #321

NegativeK wrote: post #318

Earlier quoted context omitted.

I really appreciate you giving a shit. Not sarcastically -- it seems like you're actually doing everything right, and it makes a difference. Gating robots.txt might be a mistake, but it also might be a quick way to deal with crawlers who mine robots.txt for pages that are more interesting. It's also a page that's never visited by humans. So if you make it a tarpit, you both refuse to give the bot more information and…

> It's also a page that's never visited by humans. Never is a strong word. I have definitely visited robots.txt of various websites for a variety of random reasons. - remembering the format - seeing what they might have tried to "hide" - using it like a site's directory - testing if the website is working if their main dashboard/index is offline

Are you sure you are human?

Re: Ban me at the IP level if you don't like me

#329

Post by nailer » Mon, Aug 25, 2025, 4:25 PM UTC

alistairSH wrote: post #272

nailer wrote: post #213

Earlier quoted context omitted.

I’m not claiming everyone pronounces it that way. But he’s an ero, we need to find an ospital, ninety miles an our. You will find government documents and serious newspapers that refer to an hospital.

Generic American English pronounces the 'h' in hospital, hero, heroine, but not hour. Same is true for RP English. Therefore, for both accents/dialects, the correct phrases are "a hotel", "a hero", "a heroine", and "an hour". Cockney, West Country, and a few other English accents "h drop" and would use "an 'our", "an 'otel", etc.

> RP English

One might think RP English certainly doesn't determine correctness.

Re: Ban me at the IP level if you don't like me

#330

Post by zipliner » Mon, Aug 25, 2025, 4:33 PM UTC

McDyver wrote: post #242

zipliners wrote: post #84

Earlier quoted context omitted.

Allow/deny list is more descriptive. That's one good reason for using those terms. Do you agree? In reply to your argument, the deny list (the actual list, apart from what term we use for it) is necessarily something negatively laden, since the items denied are denied due to the real risks/costs they otherwise impose. So using and embracing the less direct phrase 'black' rather than 'deny' in this case seems unlikely…

I understand your point, but my argument is in the more generic aspect. Consider how whoever complains about blacklist/whitelist would eventually complain about about allow/deny and say they are non-inclusive. Where would this stop? I would say that as long as the term in unequivocal (and not meant to be offensive) in the context, then there's no need to self-censor

> would eventually

That's an empirical premise in a slippery slope style argument. Any evidence to back it up? Who is opposing the terms allow/deny and why? I don't see it.

> no need to self-censor

The terms allow/deny are more directly descriptive and less contested which I see as a clear win-win change, so I've shifted to use those terms. No biggie and I don't feel self-censored by doing so.