Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Block Some of the Bots

nochan.net

Post reply on HN
Print view View original

101–110 of 158 posts

Re: How to Block Some of the Bots

#101

Post by jolux »

razorbeamz wrote: post #93
milkshakes wrote: post #85

Earlier quoted context omitted.

i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows. is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is…

Do you have a source for those numbers?

ChatGPT has over a billion monthly active users.

Re: How to Block Some of the Bots

#102

Post by Bender »

jolux wrote: post #101
razorbeamz wrote: post #93

Earlier quoted context omitted.

Do you have a source for those numbers?

ChatGPT has over a billion monthly active users.

I am an active user of Claude but I do not live my life through that thing. I try to get it to answer questions that it does not want to answer. The only questions it seems to answer without hesitation are technical in nature. In the entire existence of my account I have started 9 chats.

More interesting to me was that Alexa on Amazon Prime answered all my questions without hesitation or disclaimers but once I have my answers I close that tab.

Re: How to Block Some of the Bots

#103

Post by JSR_FDED »

Bender wrote: post #96
JSR_FDED wrote: post #95

Earlier quoted context omitted.

I’ll tell you why this isn’t a great experience for me. I try to actively participate on HN, and based on a title I click on posts that promise to be educational or thought-provoking. The topic of combatting bots is very much on my mind. Instead of learning something I get a blank screen, for the crime of being interested and living somewhere you deem block worthy. Cool.

Try to force a refresh it may work now.

It doesn’t. But thanks for trying.

Re: How to Block Some of the Bots

#104

Post by Gigachad »

userbinator wrote: post #37

You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago. Block on traffic volume and request frequency if that's causing a problem. (And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with…

The user agent string is harmful in general. Any new browser should just copy the chrome user agent.

Re: How to Block Some of the Bots

#105

Post by mkj »

AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things.

Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine!

But getting 2.7 million requests/day from residential proxy IPs was a bit taxing for the old uwsgi/cgi programs (that's average 30/s), making other sites on the server suffer. It was also getting close to 1TB/month which is the VPS quota.

So I ended up turning them off, one more cut towards the centralised internet. Sure I could put a cache in front, but there are probably millions of combinations of dynamic urls in the VCS tree so caching mightn't help much anyway, not worth the effort. I'm sure the web server could be tuned better too, but again it's a waste of time.

Re: How to Block Some of the Bots

#106

Post by Bender »

Bender wrote: post #69
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…

As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.

I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.

Re: How to Block Some of the Bots

#107

Post by Barbing »

JSR_FDED wrote: post #95
Bender wrote: post #92

Earlier quoted context omitted.

It's crazier than that. I block two entire continents. Don't worry you are not missing anything. There is a link to archive.is that mirrors the page if you are curious.

I’ll tell you why this isn’t a great experience for me. I try to actively participate on HN, and based on a title I click on posts that promise to be educational or thought-provoking. The topic of combatting bots is very much on my mind. Instead of learning something I get a blank screen, for the crime of being interested and living somewhere you deem block worthy. Cool.

Oh I came around myself. We got a blank screen and then the author was awesome and explained themselves and I'm satisfied having this one link as an extreme blocking test. Especially because they were thoughtful enough to archive the site themselves.

Our situations to be fair aren’t equivalent. If my block were due to my region instead of an optional paid service, that would feel worse. Maybe it wouldn’t have been the first time either. Whole continents written off!

Let’s certainly loudly complain if someone implements all these methods and inadvertently blocks us when they should’ve known better or don’t care about collateral damage. (If they’re apologetic, between a rock and a hard place with receipts, would require further discussion. e.g. is their choice really between serving us and closing shop)

Edit: but title could add “(but don’t)” at the end

Re: How to Block Some of the Bots

#108

Post by Barbing »

mkj wrote: post #105
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things. Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine! But getting 2.7 million requests/day from…

30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?

Re: How to Block Some of the Bots

#109

Post by Barbing »

Bender wrote: post #106
Bender wrote: post #69

Earlier quoted context omitted.

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…

As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers. I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.

Bender, even got yourself a great username. One of the most responsive authors in memory.

Re: How to Block Some of the Bots

#110

Post by Ndymium »

AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

In my case it's my Forgejo instance. My blog is static files with a limited amount of pages to crawl and indeed it's no issue at all. Forgejo is a dynamic service with an infinite amount of pages to discover and it runs git in the background to generate (some of) the pages. Tons of bots can easily overwhelm my small server. I could hide the repositories, but they're open on purpose (it's open source after all).

I have it guarded by a simple cookie check now, which seems to work well enough. Only a small subset of bots pass it. It likely kills search engine discoverability, but that's the tradeoff I have to make.

Post reply on HN