51–60 of 88 posts
One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted with ai crawlers. I put a project online - it was online for a month, and the second I added an SSL cert it went from 0 traffic to 1000 requests/min.
Bot traffic Share of HTTP requests Ranking Location Percentage 1. Gibraltar 92.0% 2. Iran 76.9% 3. Singapore 76.4% 4. Ireland 72.9% 5. Netherlands 68.8% Lol, what is happening?
One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted with ai crawlers. I put a project online - it was online for a month, and the second I added an SSL cert it went from 0 traffic to 1000 requests/min.
Make a new certificate, let crawlers blast you and add those IPs to a block list.
One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted with ai crawlers. I put a project online - it was online for a month, and the second I added an SSL cert it went from 0 traffic to 1000 requests/min.
Make a new certificate, let crawlers blast you and add those IPs to a block list.
edit: 'nikcub beat me to it by 30 seconds :)
> Percentage of HTTP requests classified as bot (automated) or human. Filtered to HTML responses, representing web page traffic. (Emphasis mine) I realize that this is likely an inherent limitation, but there is a difference between "bot vs human traffic" and "traffic that CF thinks is bot/human". Every time CF blocks me, I assume it claims I'm a bot in this chart.
Bot traffic Share of HTTP requests Ranking Location Percentage 1. Gibraltar 92.0% 2. Iran 76.9% 3. Singapore 76.4% 4. Ireland 72.9% 5. Netherlands 68.8% Lol, what is happening?
Gib probably has a handful of servers scraping, but the place is so small, it almost it eclipses normal traffic
The geology of the area seems to make for good cooling and DR sturdiness. One DC is even 500 meters under rock.
https://www.datacentermap.com/gibraltar/gibraltar/
https://www.datacentermap.com/gibraltar/gibraltar/continent8...
Bot traffic Share of HTTP requests Ranking Location Percentage 1. Gibraltar 92.0% 2. Iran 76.9% 3. Singapore 76.4% 4. Ireland 72.9% 5. Netherlands 68.8% Lol, what is happening?
Gib probably has a handful of servers scraping, but the place is so small, it almost it eclipses normal traffic
Bot traffic Share of HTTP requests Ranking Location Percentage 1. Gibraltar 92.0% 2. Iran 76.9% 3. Singapore 76.4% 4. Ireland 72.9% 5. Netherlands 68.8% Lol, what is happening?
I do understand why all of those bot scores are so high. But netherlands? Are there big datacenter providers I dont know of that are used by bots?
> Percentage of HTTP requests classified as bot (automated) or human. Filtered to HTML responses, representing web page traffic. (Emphasis mine) I realize that this is likely an inherent limitation, but there is a difference between "bot vs human traffic" and "traffic that CF thinks is bot/human". Every time CF blocks me, I assume it claims I'm a bot in this chart.
Cloudflare are more likely to be undercounting bots - they don't really pick up many of the modern browser-driven bots and crawlers.
> Percentage of HTTP requests classified as bot (automated) or human. Filtered to HTML responses, representing web page traffic. (Emphasis mine) I realize that this is likely an inherent limitation, but there is a difference between "bot vs human traffic" and "traffic that CF thinks is bot/human". Every time CF blocks me, I assume it claims I'm a bot in this chart.