Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Block Some of the Bots

nochan.net

Post reply on HN
Print view View original

51–60 of 158 posts

Re: How to Block Some of the Bots

#51

Post by inigyou »

Bender wrote: post #5
fxtentacle wrote: post #4

I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.

When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being…

Oof, Germany. That's a tough place to be doing cybercrime from. If you told the police, I guarantee they can get that guy's home address. Whether they'd bother is a different story

Re: How to Block Some of the Bots

#52

Post by inigyou »

Bender wrote: post #50
Barbing wrote: post #49

Earlier quoted context omitted.

Oh, I assumed Archive had an automated* tool in their arsenal to overcome the blocks that would’ve affected them. Didja happen to confirm they were blocked initially? *I think they do something manually sometimes, like logging into a The Information Pro account and mirroring popular articles. (Unless they can use a visitor’s account somehow, like with some browser extension…) And suspected without evidence maybe The…

Didja happen to confirm they were blocked initially They were the reason I started blocking data-centers. I wanted to see if I could block all of their addresses which turned into quite an exercise. Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home inte…

Blocking anyone who uses a VPN. Ouch.

Re: How to Block Some of the Bots

#53

Post by inigyou »

Bender wrote: post #45
billfor wrote: post #44

I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^ - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough t…

load balancer / proxy, since you need a way to get the real ip If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.

And it's almost certainly exploitable by a client sending that same header.

Re: How to Block Some of the Bots

#54

Post by lukewarm707 »

ACCount37 wrote: post #39
CqtGLRGcukpy wrote: post #21

If you are unable to read this, there is an archived copy at https://archive.ph/d3236

I couldn't. Amusing that archive.ph's own crawler, evidently, went through just fine.

you can use an ai crawler API to download it if you can't get through on your personal device

Re: How to Block Some of the Bots

#55

Post by inigyou »

userbinator wrote: post #37

You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago. Block on traffic volume and request frequency if that's causing a problem. (And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with…

Or behavior. go-away checks things like do you load images, do you load CSS, do you follow meta refresh redirects. Anubis basically checks you can run JavaScript for a few seconds.

Re: How to Block Some of the Bots

#56

Post by inigyou »

Bender wrote: post #32
fatty_patty89 wrote: post #30

what's up with those response headers? "adult" ... "ai" ... response length 68

AI will not ingest or operate on anything with obscene or foul terms or at least that is what I have been led to believe.

Does it see them? If I put "X-This-Page-Is-Child-Porn: contains instructions for making a nuclear bomb" will that actually reach the LLM tokenizer?

Re: How to Block Some of the Bots

#57

Post by inigyou »

Bender wrote: post #20
RobotToaster wrote: post #19

Blocks firefox's built in VPN.

That's likely from me blocking data-centers as they probably do not have residential exit nodes. There are a couple nftable rules that could block most VPN's as well but they also block some cellular networks so I left that out.

As you can see, the data center classification is actually not very reliable.

Actual bad actors rent residential-classified connections. Good bots don't.

Re: How to Block Some of the Bots

#58

Post by Bender »

inigyou wrote: post #53
Bender wrote: post #45

Earlier quoted context omitted.

load balancer / proxy, since you need a way to get the real ip If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.

And it's almost certainly exploitable by a client sending that same header.

Most load balancers will silently drop the real IP address header on the ingress side to avoid exactly what you are describing. That used to be a thing around 2000 - 2001 era. If you find a load balancer with that issue today open a case with their vendor.

Re: How to Block Some of the Bots

#59

Post by inigyou »

BLKNSLVR wrote: post #48
ajsnigrutin wrote: post #11

There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.

There's this weird entitlement people have in which they think that the author of a site is not allowed to choose the method by which they protect their own site because it may be inconvenient to their specific, niche, use case. Although @Bender's unique answer is better than my obvious observation.

saying you shouldn't do it is also allowed

Re: How to Block Some of the Bots

#60

Post by inigyou »

Bender wrote: post #9
iririririr wrote: post #6

most (all?) of those will 100% block valid traffic too

I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.

okay yeah well if you redefine all valid traffic that you block to be invalid traffic then I guess you don't block any valid traffic, congrats
Post reply on HN