Earlier quoted context omitted.
Do you have a source for those numbers?
ChatGPT has over a billion monthly active users.
This does not mean that all billion of them are using ChatGPT as their main interface with the internet.
How to Block Some of the Bots
121–130 of 158 posts
Re: How to Block Some of the Bots
#122Earlier quoted context omitted.
30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?
> Are these loser companies who don’t want to cache on their end Google couldn't be bothered to cache; I would guess this before anything else. https://drewdevault.com/blog/Google-has-been-DDoSing-sourceh...
> each IP address still clones the same repositories 8-10 times per hour.
Dumb! When was this fixed and don’t say never :)
Re: How to Block Some of the Bots
#123I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…
My dedicated server comes with 2TB/month data included. Humans use up about 2% of that. If I didn't block bots I'd run out mid-month.
Re: How to Block Some of the Bots
#124Earlier quoted context omitted.
For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…
i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows. is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is…
I wonder what the click through rate is for the source links. Based on what I've been hearing, it's very low. And that's if the particular LLM/interface/answer even bothers to add them.
If the bots are eating your site traffic and users, what's the point of allowing them in?
Re: How to Block Some of the Bots
#125Earlier quoted context omitted.
i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows. is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is…
I like those ideas. If people are truly operating purely through LLM's then I am fine with being partitioned from them. The LLM operators will provide their reality and their truth. This is all for low trust internet-wide access. For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on for…
i guess what i'm curious about is where you would draw the line, and why.
how do you define a user agent? is it desirable or not for users to be able to discover and access resources and communities on the internet using the tools, formats, and workflows that they prefer?
are search bots desirable?
the internet wayback machine?
how do you feel about browser extensions and greasemonkey scripts?
Re: How to Block Some of the Bots
#126I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…
One of my issues had to do with pure bandwidth, dozens of bots were requesting the same image over and over again adding random crap to the query. My server was being slow for me during crawling peaks.
> An actual proper static site would likely 10x that.
A lot of the stuff I host isn't static. Any non-static content also gets overloaded pretty quickly. Hosting something like a Git server can turn pretty resource intensive if shitty bots start diffing every commit with every other commit.
I'm generally lazy when it comes to blocking IPs, but when I do run into an issue, I just blanket block IP ranges. When that doesn't work, ASNs get blocked. So far I haven't had to take most measures mentioned here but I probably will once the broken bots start coming from botnets using residential IP addresses.
Re: How to Block Some of the Bots
#127Earlier quoted context omitted.
saying you shouldn't do it is also allowed
So long as a some half-decent explanation of reasoning is provided. Plenty of people would say that I shouldn't block residential proxies, but I would 100% block residential proxies if I could. I think most people responsible for blocking traffic would actively manage their rules if it was costing them something valuable. If they don't notice, then, well no harm no foul.
That's a long list of things you want to block. Can you even name what you don't want to block?
Re: How to Block Some of the Bots
#128Earlier quoted context omitted.
Not all residential proxies are malware. You can get paid crypto to offer up your connection. Prices will go up because of the malware crackdowns but the industry will still exist.
> You can get paid crypto to offer up your connection. Voluntary malware? Funded malware facilitation? I'm not against crypto, but it would sound more legit if you removed the word crypto from your sentence. If it's exclusively the ability to be paid in crypto, as opposed to local currency or USD, then I'd guess it leans pretty heavily to shady-ness. I pretty much think all residential proxy traffic is undesirable -…
What part of that is malware?
We've already established that you wish to block basically the entire internet including two actual continents, no wonder someone who was really interested in your site would try to view it with a residential proxy.
How about phonecalling someone with an unblocked connection and having them send pictures of the pages. Would you block that if you could?
Re: How to Block Some of the Bots
#129Earlier quoted context omitted.
It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things. Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine! But getting 2.7 million requests/day from…
30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?
I have compiled a list of vibecoders who have impressed me with their consideration for other people. Here it is:
Re: How to Block Some of the Bots
#130Earlier quoted context omitted.
Most load balancers will silently drop the real IP address header on the ingress side to avoid exactly what you are describing. That used to be a thing around 2000 - 2001 era. If you find a load balancer with that issue today open a case with their vendor.
Not reliably enough. Look for parser differentials.
Please enlighten us