Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Block Some of the Bots

nochan.net

Post reply on HN
Print view View original

91–100 of 158 posts

Re: How to Block Some of the Bots

#91

Post by aorth »

inigyou wrote: post #62
ferngodfather wrote: post #16

Earlier quoted context omitted.

That's fair enough. I think it's a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often "use Cloudflare" is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.

Cheap and quick bot-blocking is provided by go-away, less cheap and quick by Anubis.

Sadly, go-away https://git.gammaspectra.live/git/go-away doesn't seem to be maintained anymore.

Re: How to Block Some of the Bots

#92

Post by Bender »

JSR_FDED wrote: post #89
Bender wrote: post #29

Earlier quoted context omitted.

That makes sense. I block most data-centers and that would include much of Apple's data-centers. What I am doing is of course unorthodox for a blog. This is more of a test site to show what could be done. People can pick and choose which features they like for their particular use cases. I could some day split it out into a blog and a demo site.

No access from residential IP in Singapore. Why bother to get all sophisticated about blocking, just power down your server.

It's crazier than that. I block two entire continents. Don't worry you are not missing anything. There is a link to archive.is that mirrors the page if you are curious.

Re: How to Block Some of the Bots

#93

Post by razorbeamz »

milkshakes wrote: post #85
Bender wrote: post #69

Earlier quoted context omitted.

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…

i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows. is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is…

Do you have a source for those numbers?

Re: How to Block Some of the Bots

#95

Post by JSR_FDED »

Bender wrote: post #92
JSR_FDED wrote: post #89

Earlier quoted context omitted.

No access from residential IP in Singapore. Why bother to get all sophisticated about blocking, just power down your server.

It's crazier than that. I block two entire continents. Don't worry you are not missing anything. There is a link to archive.is that mirrors the page if you are curious.

I’ll tell you why this isn’t a great experience for me. I try to actively participate on HN, and based on a title I click on posts that promise to be educational or thought-provoking. The topic of combatting bots is very much on my mind.

Instead of learning something I get a blank screen, for the crime of being interested and living somewhere you deem block worthy. Cool.

Re: How to Block Some of the Bots

#96

Post by Bender »

JSR_FDED wrote: post #95
Bender wrote: post #92

Earlier quoted context omitted.

It's crazier than that. I block two entire continents. Don't worry you are not missing anything. There is a link to archive.is that mirrors the page if you are curious.

I’ll tell you why this isn’t a great experience for me. I try to actively participate on HN, and based on a title I click on posts that promise to be educational or thought-provoking. The topic of combatting bots is very much on my mind. Instead of learning something I get a blank screen, for the crime of being interested and living somewhere you deem block worthy. Cool.

Try to force a refresh it may work now.

Re: How to Block Some of the Bots

#97

Post by loloquwowndueo »

AussieWog93 wrote: post #84
loloquwowndueo wrote: post #79

Earlier quoted context omitted.

> WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Yes we know - I still had my site knocked out a couple of times by overeager bots. They can be really abusive.

Wow, that's nuts. What sort of site was this? Mine don't get anything near this level of attention!

Just a personal blog, and I have to admit it had a misconfiguration that was my fault - nonexistent urls would redirect to the index. Still, dumb scraper bots would 1) not realize there was a redirect loop and kept trying every possible url, and 2) they were doing so aggressively enough that it brought the site down a couple of times. A well-behaved crawler would have slowed down and noticed the loop, never had issues with crawlers for years until the advent of ai.

Re: How to Block Some of the Bots

#98

Post by Bender »

AussieWog93 wrote: post #83
Bender wrote: post #69

Earlier quoted context omitted.

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…

Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.

Thrill of the game is a great way to put it.

Re: How to Block Some of the Bots

#99

Post by Bender »

Bender wrote: post #26
genodethrowaway wrote: post #24

and all valid traffic too, judging by these HN comments (and my own attempts to connect).

The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.

As of now roughly 4385 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.

Re: How to Block Some of the Bots

#100

Post by cuchoi »

AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

Hundreds of fake applicants to a position
Post reply on HN