Live data from Hacker News

Bot vs human traffic

radar.cloudflare.com

71–80 of 88 posts

Re: Bot vs human traffic

#71

Any thoughts on why ~30% of HTTP request are in US? I know we had first mover advantage for awhile but I'd expect this to have been diluted by larger populations by now. It doesn't appear to be AI/bot driven either.

Network effect feedback. Cheap hosting in the US because servers are there, more servers are there because of demand for hosting. AWS is there - similar reasons. Big Tech had more time to develop there and eclipsed other countries' tech.

Re: Bot vs human traffic

#73
post #52

Earlier quoted context omitted.

I do understand why all of those bot scores are so high. But netherlands? Are there big datacenter providers I dont know of that are used by bots?

Amsterdam is one of the largest global internet hubs.

https://qde.com/blog/what-is-ams-ix/

Re: Bot vs human traffic

#74

One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted with ai crawlers. I put a project online - it was online for a month, and the second I added an SSL cert it went from 0 traffic to 1000 requests/min.

Today AI crawlers, years ago vulnerability scanners from Russia or China.

Either way! People monitor cert registries for targets.

Re: Bot vs human traffic

#75

One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted with ai crawlers. I put a project online - it was online for a month, and the second I added an SSL cert it went from 0 traffic to 1000 requests/min.

> One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert I've been thinking of using wildcard certs for Caddy in regards to this.

and then what? serve your app under some obscure / customer unfriendly subdomain?

Re: Bot vs human traffic

#76

Earlier quoted context omitted.

Why would blocking those be a problem?

because you are blocking all of Comcast, Verizon, T-Mobile, British Telecom, .... at the end you have blocked every network with human visitors and only datacenter IPs can access your site. The proxies rotate IP every day, so you either have ineffective blocking or you block the whole network.

My site is not for americans so I don't care about blocking american isps

Re: Bot vs human traffic

#77
post #64

Earlier quoted context omitted.

Why would blocking those be a problem?

there are 150M+ of them and you'll be taking out a lot of human users with it modern blocking is behaviour / heuristic based

There are 150 million bots all using residential IP addresses?

Re: Bot vs human traffic

#78

Earlier quoted context omitted.

> One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert I've been thinking of using wildcard certs for Caddy in regards to this.

and then what? serve your app under some obscure / customer unfriendly subdomain?

Even if you use a common subdomain, anecdotally I get orders of magnitude less bot traffic than not using a wildcard cert.

Re: Bot vs human traffic

#79
post #18

It's a silly metric. There could be only one master bot that pings every known endpoint multiple times a second, and that would probably surpass all human activity, too. It doesn't really tell us much about intention or the ability to masquerade as humans. Where I would start to worry is if there's evidence that bot access patterns are starting to become harder to distinguish from human access patterns, which would s…

>but I do worry about how many social bots are attempting to manipulate or mislead people. You should browse reddit sometime. The easy ones to spot just autocreate accounts using the autoname at signup, which is of the formfactor [word1][word2]/d{4} Regex nazis please spare me, I am doing my bestest

..... I like my auto generated username it's a funny one
Post reply on HN