Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Block Some of the Bots

nochan.net

Post reply on HN
Print view View original

81–90 of 158 posts

Re: How to Block Some of the Bots

#81

Post by strken »

Bender wrote: post #13
ajsnigrutin wrote: post #11

There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.

A guy gets sent to hell. The demons guide him into the lobby. Satan jumps out and lets out a big evil roar to no effect. Satan then glances down and sees the spot on the mans finger where there was a ring. "Oh... well you've been through worse. The break room is down the hallway to the left, mail room is upstairs to the right..." Satan just walks away. Every step is optional of course. There are ways to make curl wor…

Huh. I should probably add a .curlrc file with a generic Chrome UA and see if it can get past the most annoying filters.

This has always struck me as dishonest, but recently I've been blocked from accessing the docs of a service I pay for.

Re: How to Block Some of the Bots

#82

Post by BLKNSLVR »

inigyou wrote: post #59
BLKNSLVR wrote: post #48

Earlier quoted context omitted.

There's this weird entitlement people have in which they think that the author of a site is not allowed to choose the method by which they protect their own site because it may be inconvenient to their specific, niche, use case. Although @Bender's unique answer is better than my obvious observation.

saying you shouldn't do it is also allowed

So long as a some half-decent explanation of reasoning is provided.

Plenty of people would say that I shouldn't block residential proxies, but I would 100% block residential proxies if I could.

I think most people responsible for blocking traffic would actively manage their rules if it was costing them something valuable. If they don't notice, then, well no harm no foul.

Re: How to Block Some of the Bots

#83

Post by AussieWog93 »

Bender wrote: post #69
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…

Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.

Re: How to Block Some of the Bots

#84

Post by AussieWog93 »

loloquwowndueo wrote: post #79
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

> WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Yes we know - I still had my site knocked out a couple of times by overeager bots. They can be really abusive.

Wow, that's nuts. What sort of site was this? Mine don't get anything near this level of attention!

Re: How to Block Some of the Bots

#85

Post by milkshakes »

Bender wrote: post #69
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…

i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows.

is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is the strategy?

Re: How to Block Some of the Bots

#86

Post by Bender »

milkshakes wrote: post #85
Bender wrote: post #69

Earlier quoted context omitted.

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…

i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows. is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is…

I like those ideas. If people are truly operating purely through LLM's then I am fine with being partitioned from them. The LLM operators will provide their reality and their truth. This is all for low trust internet-wide access.

For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on forums and such. That is where strategy comes into play and has been working well for some time.

Re: How to Block Some of the Bots

#87

Post by charcircuit »

Bender wrote: post #5
fxtentacle wrote: post #4

I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.

When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being…

>to figure out which records to avoid

It is more likely they were looking for a DNS server with a big response packet for use in amplified DDoS attacks.

Re: How to Block Some of the Bots

#88

Post by extra88 »

AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

I have a personal site on a shared host that was recently suspended for excessive CPU use caused by incessant AI bot crawling. I don't have a problem with crawling per se, there's just so many and they're so bad at it.

Re: How to Block Some of the Bots

#89

Post by JSR_FDED »

Bender wrote: post #29
Barbing wrote: post #27

Earlier quoted context omitted.

Thanks. Regretful implementation*, being a normie on iOS Safari: https://i.ibb.co/vCDH79d0/IMG-0303.png …and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.) *edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmin…

That makes sense. I block most data-centers and that would include much of Apple's data-centers. What I am doing is of course unorthodox for a blog. This is more of a test site to show what could be done. People can pick and choose which features they like for their particular use cases. I could some day split it out into a blog and a demo site.

No access from residential IP in Singapore. Why bother to get all sophisticated about blocking, just power down your server.

Re: How to Block Some of the Bots

#90

Post by Bender »

charcircuit wrote: post #87
Bender wrote: post #5

Earlier quoted context omitted.

When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being…

>to figure out which records to avoid It is more likely they were looking for a DNS server with a big response packet for use in amplified DDoS attacks.

No they were specifically trying and failing of course to AXFR from the only domains I had added the zeroconf addresses to and shortly thereafter all the scanning went quiet. If they were not skiddies they could have just updated their tools to ignore private address spaces from all domains all together. Don't worry, I am not reporting them.

There are loads of big TXT records on many domains that can be used for amplification attacks. [1] These are just a few by the way, I transparently filter many TXT records on some DNS servers for this reason.

[1] - https://news.ycombinator.com/item?id=48599363

Post reply on HN