Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ban me at the IP level if you don't like me

boston.conman.org

Post reply on HN
Print view View original

301–310 of 516 posts

Re: Ban me at the IP level if you don't like me

#301

Post by brianwawok »

8organicbits wrote: post #235

I've been working on a web crawler and have been trying to make it as friendly as possible. Strictly checking robots.txt, crawling slowly, clear identification in the User Agent string, single IP source address. But I've noticed some anti-bot tricks getting applied to the robot.txt file itself. The latest was a slow loris approach where it takes forever for robots.txt to download. I accidentally treated this as a 404…

I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother downloading it.

Never assume malice what can be attributed to incompetence.

Re: Ban me at the IP level if you don't like me

#302

Post by blueflow »

praptak wrote: post #97

"I'm seriously thinking that the CCP encourage this with maybe the hope of externalizing the cost of the Great Firewall to the rest of the world. If China scrapes content, that's fine as far as the CCP goes; If it's blocked, that's fine by the CCP too (I say, as I adjust my tin foil hat)." Then turn the tables on them and make the Great Firewall do your job! Just choose a random snippet about illegal Chinese occupati…

I just tried this, i took some strings about Falun Gong and the Tianmen thing from the chinese wikipedia and put them into my SSH server banner. The connection attempts from the Tencent AS ceased completely, but now they come from Russia, Lithuania and Iran instead.

Re: Ban me at the IP level if you don't like me

#303

Post by kiitos »

phito wrote: post #134
bob1029 wrote: post #132

I think a lot of really smart people are letting themselves get taken for a ride by the web scraping thing. Unless the bot activity is legitimately hammering your site and causing issues (not saying this isn't happening in some cases), then this mostly amounts to an ideological game of capture the flag. The difference being that you'll never find their flag. The only thing you win by playing is lost time. The best wa…

My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it does not impact his service, at the very least it feels like harassment

every single IPv4 address in existence receives constant malicious traffic, from uncountably many malicious actors, on all common service ports (80, 443, 22, etc.) and, for HTTP specifically, to an enormous and growing number of common endpoints (mostly WordPress related, last I checked)

if you put your server up on the public internet then this is just table stakes stuff that you always need to deal with, doesn't really matter whether the traffic is from botnets or crawlers or AI systems or anything else

you're always gonna deal with this stuff well before the requests ever get to your application, with WAFs or reverse proxies or (idk) fail2ban or whatever else

also 1000 req/hour is around 1 request every 4 seconds, which is statistically 0 rps for any endpoint that would ever be publicly accessible

Re: Ban me at the IP level if you don't like me

#304

Post by hvb2 »

JohnFen wrote: post #244
Thorrez wrote: post #221

Earlier quoted context omitted.

What is beyond irresponsible? Monitoring logs and researching odd things found there?

How are passwords ending up in your logs? Something is very, very wrong there.

If the caller puts it in the query string and you log that? It doesn't have to be valid in your application to make an attacker pass it in.

So unless you're not logging your request path/query string you're doing something very very wrong by your own logic :). I can't imagine diagnosing issues with web requests and not be given the path + query string. You can diagnose without but you're sure not making things easier

Re: Ban me at the IP level if you don't like me

#305

Post by stronglikedan »

wvbdmp wrote: post #148
dmesg wrote: post #141

Earlier quoted context omitted.

Yes and it makes reading your logs needlessly harder. Sometimes I find an odd password being probed, search for it on the web and find an interesting story, that a new backdoor was discovered in a commercial appliance. In that regard reading my logs led me sometimes to interesting articles about cyber security. Also log flooding may result in your journaling service truncating the log and you miss something important…

You log passwords?

Sure, why not. Log every secret you come across (or that comes across you). Just don't log your own secrets. Like OP said, it lead down some interesting trails.

Re: Ban me at the IP level if you don't like me

#306

Post by JohnFen »

socksy wrote: post #289
dmesg wrote: post #255

Earlier quoted context omitted.

Does an attacking bot know your webserver is not a misconfigured router exposing its web interface to the net? I often am baffled what conclusions people come up with from half reading posts. I had bots attack me with SSH 2.0 login attempts on port 80 and 443. Some people underestimate how bad at computer science some skids are.

Also baffled that three separate people came to that conclusion. Do they not run web servers on the open web or something? Script kiddies are constantly probing urls, and urls come up in your logs. Sure it would be bad if that was how your app was architected. But it's not how it's architected, it's how the skids hope your app is architected. It's not like if someone sends me a request for /wp-login.php that my rails…

> Do they not run web servers on the open web or something?

Until AI crawlers chased me off of the web, I ran a couple of fairly popular websites. I just so rarely see anybody including passwords in the URLs anymore that I didn't really consider that as what the commenter was talking about.

Re: Ban me at the IP level if you don't like me

#307

Post by JdeBP »

ralferoo wrote: post #265
phito wrote: post #134

Earlier quoted context omitted.

My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it does not impact his service, at the very least it feels like harassment

What's worse is when you get bots blasting HTTP traffic at every open port, even well known services like SMTP. Seriously, it's a mail server. It identified itself as soon as the connection was opened, if they waited 100ms-300ms before spamming, they'd know that it wasn't HTTP because the other side wouldn't send anything at all if it was. There's literally no need to bombard a mail server on a well known port by con…

I remember putting dummy GET/PUT/HEAD/POST verbs into SMTP Relay softwares a quarter of a century ago. Attackers do not really save themselves time and money by being intelligent about this. So they aren't.

There are attackers out there that send SIP/2.0 OPTIONS requests to the GOPHER port, over TCP.

Re: Ban me at the IP level if you don't like me

#309

Post by Arnavion »

rnhmjoj wrote: post #44
ta8645 wrote: post #25

If ipv6 ever becomes a thing, it'll make blocking all that much harder.

No, it's really the same thing with just different (and more structured) prefix lengths. In IPv4 you usually block a single /32 address first, then a /24 block, etc. In IPv6 you start with a single /128 address, a single LAN is /64, an entire site is usually /56 (residential) or /48 (company), etc.

Note that for the sake of blocking internet clients, there's no point blocking a /128. Just start at /64. Blocking a /128 is basically useless because of SLAAC.

Re: Ban me at the IP level if you don't like me

#310

Post by sneak »

worthless-trash wrote: post #93
sneak wrote: post #79

I feel like people seem to forget that an HTTP request is, after all, a request . When you serve a webpage to a client, you are consenting to that interaction with a voluntary response. You can blunt instrument 403 geoblock entire countries if you want, or any user agent, or any netblock or ASN. It’s entirely up to you and it’s your own server and nobody will be legitimately mad at you. You can rate limit IPs to x re…

> This whole AI scraper panic is so incredibly overblown. The problem is that its eating into peoples costs, and if you're not concerned with money, I'm just asking, can you send me $50.00 USD ?

If people don’t want to spend the money serving the requests, then their own servers are misconfigured because responding is optional.
Post reply on HN