141–150 of 158 posts
I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…
Earlier quoted context omitted.
It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things. Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine! But getting 2.7 million requests/day from…
> 1TB/month Switch VPS providers, you can easily find higher numbers or "unlimited".
Earlier quoted context omitted.
The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
As of now roughly 4385 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.
the real humans you are seeing, are bots.
brightdata is one vendor, they have 150m genuine residential ips, aquired through standard business. actually, since this was kind of an invitation, i scraped it. the volume is so small compared to the ai corpus, this volume does not cost anything. as i suspected it was a 100% success rate. the only page it did not reach was the one needing authentication. the tests, they were fine. of course, ai knows ascii from this type of data but it wouldn't be able to read the page unless you use a screenshot.
the bots are not interested in the headers. they never see them. imagine a normal person at home using their browser, that's the bot. the reason this is the case is because the bot ip is someone's house (at least for premium proxy), and the browser fingerprint is a recording of someone's browser. oversimplified of course.
the 'bots' are the public facing corpo scrapers, and the 'users' are bots run by data vendors.
to test this, there is a ranking on 'foil' bot detection. to actually detect them they need 0.05 usd per check, and they are more interested in fraud via automated interaction and account creation.
grab a free account, you can very easily collect such data.
Earlier quoted context omitted.
For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript . Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any…
As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers. I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.
You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP. Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers. > One class of bot…
I will keep the blocking of http/1.1 at the risk of blocking ancient browsers and API tools. That's just too easy for people to update these days and they are not missing much by not visiting cranky ol' me. If we were talking about proprietary code running in an old financial system I would totally get it, but the public internet can and should update for their own sake.
Any request coming to me claiming to be from Google I know is a lie as I have been blocking Google for a fairly long time. If anything Google would say I am malicious for blocking them, or at least they used to. I cycle by blog through many random domains to break any associations and related snapshots to disrupt birds of a feather associations. I try to control how people discover my silly ramblings.
Earlier quoted context omitted.
I like those ideas. If people are truly operating purely through LLM's then I am fine with being partitioned from them. The LLM operators will provide their reality and their truth. This is all for low trust internet-wide access. For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on for…
i guess what i'm curious about is where you would draw the line, and why. how do you define a user agent? is it desirable or not for users to be able to discover and access resources and communities on the internet using the tools, formats, and workflows that they prefer? are search bots desirable? the internet wayback machine? how do you feel about browser extensions and greasemonkey scripts?
I have no need for search bots personally. If I had a complex site I would build my own search feature. If it was a commercial site I would pay for advertising on popular sites that were of related categories.
Wayback is fun to play with but it's just a toy to me. There is no concept of domain ownership. Most of the snapshots of the domain I am using were from when someone else rented it. They claim to respect robots.txt but that is not entirely true. They crawl and save content even if a site says not to and as soon as the site is offline and robots.txt is no longer accessible they will display all the archived content. This means a person has to park their domain on a server containing a robots.txt that matches their intentions.
Browser extensions are mostly invisible. What is obvious to me is when people are using "reader" applications as web clients. They should be concerned more than me. They are reading random sites run by strangers using apps that may or may not have been battle hardened, reviewed by third party penetration testers and so on. Some of them are vibe coded in unsafe languages.
I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…
You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP. Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers. > One class of bot…
I also got a free pentest. I've expanded since. Takeaways: my mitigations and pipeline are solid. If the people on HN pleading for the poor LLM users and google search have any relationship with the people providing the free pentest... I mean, they don't? Srsly.
As for blocking IPs... the survival pressure has forced 90% of the bot traffic to VPNs. Man, they're lighting up the VPN endpoints like christmas trees. Little points of light, very pretty. Thanks for the fish. I could make feeds of the one-shot wonders available, my infra is solid, you'd need to be properly vetted and I'd need to approve the use case.
This is fun.
You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP. Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers. > One class of bot…
Blocking http1.1 is reasonable imo, its been over a decade since just about every browser has supported it. Anyone on a browser that old has most the mainstream web broken to them at this point, a broken (personal) site wouldnt be an exception to them, but the norm.