61–70 of 88 posts
Earlier quoted context omitted.
Make a new certificate, let crawlers blast you and add those IPs to a block list.
these old network security techniques don't really work anymore. the common bots are at known IP ranges, the problem bots are all on datacenter + residential proxies.
> Percentage of HTTP requests classified as bot (automated) or human. Filtered to HTML responses, representing web page traffic. (Emphasis mine) I realize that this is likely an inherent limitation, but there is a difference between "bot vs human traffic" and "traffic that CF thinks is bot/human". Every time CF blocks me, I assume it claims I'm a bot in this chart.
I do sometimes get blocked as a bot. I have no idea how many false positives there are, but there are some and CF does assume there are none in all their numbers (e.g. email saying they block x bots).
Earlier quoted context omitted.
these old network security techniques don't really work anymore. the common bots are at known IP ranges, the problem bots are all on datacenter + residential proxies.
Why would blocking those be a problem?
modern blocking is behaviour / heuristic based
Given how many rounds of captchas I have to fight through, I'm not sure if these numbers are accurate.
Funny how I get captcha looped with my adblocking in firefox but you can just get through easily with a few puppeteer plugins controlling headless chrome.
One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted with ai crawlers. I put a project online - it was online for a month, and the second I added an SSL cert it went from 0 traffic to 1000 requests/min.
I've been thinking of using wildcard certs for Caddy in regards to this.
Earlier quoted context omitted.
these old network security techniques don't really work anymore. the common bots are at known IP ranges, the problem bots are all on datacenter + residential proxies.
Why would blocking those be a problem?
at the end you have blocked every network with human visitors and only datacenter IPs can access your site.
The proxies rotate IP every day, so you either have ineffective blocking or you block the whole network.
Bot traffic Share of HTTP requests Ranking Location Percentage 1. Gibraltar 92.0% 2. Iran 76.9% 3. Singapore 76.4% 4. Ireland 72.9% 5. Netherlands 68.8% Lol, what is happening?
I do understand why all of those bot scores are so high. But netherlands? Are there big datacenter providers I dont know of that are used by bots?