211–220 of 228 posts
This kind of thing can be mitigated by not publishing a page/download for every single branch, commit and diff in a repo. Make only the HEAD of each branch available. Anyone who wants more detail has to clone it and view it with their favourite git client. For example https://mitxela.com/projects/web-git-sum ( https://git.mitxela.com/ )
I suspect bog-standard per-IP rate limiting would also mitigate this, no?
Is there a way to block it by shibboleth? Curious, since the recent Google hack where you add -(n-word) to the end of your query so the AI automatically shuts down works like a charm.
I have no doubt that it works and it's hilarious that it works, but is there a way that does not involve my Google search history look like I've applied for a KKK membership?
I "solved" this by adding a fail2ban rule for everyone accessing specific commits (no one does that 3 times in a row) and then blocking the following ASs completely (just too many IPs coming from those, feel free to look them up yourself): 136907 23724 9808 4808 37963 45102. And after that: sweet silence. How to block ASs? Just write a small script that queries all of their subnets once (even if it changes, its not s…
- 136907 Huawei - 23724 China Telco - 9808 China Mobile - 4808 China Unicom - 37963 Alibaba - 45102 Alibaba tech You may want to add this list as well: https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...
Earlier quoted context omitted.
Yuk… http { # ... other http settings limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s; # ... } server { # ... other server settings location / { limit_req zone=mylimit burst=20 nodelay; # ... proxy_pass or other location-specific settings } } Rate limit read-only access at the very least. I know this is a hard problem for open source projects that have relied on web access like this for a while. Anubis…
Easier said than done, I have 700k requests from bots in my access.log coming from 15k different IP addresses. :: ~/website ‹master*› » rg '(GPTBot|ClaudeBot|Bytespider|Amazonbot)' access.log | awk '{print $1}' | sort -u | wc -l 15163
map $http_user_agent $uatype {
default 'user';
~*(googlebot|bingbot) 'good_bot';
~*(nastybot|somebadscraper) 'bad_bot';
}
I’m not dismissing the difficulty of the problem but there are multiple vectors that can identify these ‘bad_bots’.
I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... 1. Anubis is a miracle. 2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t…
Earlier quoted context omitted.
> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page loa…
Oh please. That ship has sailed. I'm marginally sympathetic to people who don't run JavaScript on their browsers for a variety of reasons, but they've deliberately opted out of the de facto modern web. JS is as fundamental to current design as CSS. If you turn it off, things might work, but almost no one is testing that setup, nor should they reasonably be expected to. This has zero to do with Adtech for 99.99% of us…
99.9999% of websites shouldn't be apps in the first place.
Anyway have fun fighting AI bots and enshittifying your site in the process. In your case, I'm sure the both operators will have as much sympathy for your plight as you show others.
Of course this relies on the obscurity effect.
I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... 1. Anubis is a miracle. 2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t…
Earlier quoted context omitted.
I have no doubt that it works and it's hilarious that it works, but is there a way that does not involve my Google search history look like I've applied for a KKK membership?
`-fuck` works just as well. i guess using the n-word is just a telling and personal choice of the OP.