Viewing profile — gavinhking

Yup, in fact most of them are already. That's one of the ways this data is verifying whether the visits are spoofed or not: https://knownagents.com/insights#spoofing-and-security

Appreciate it, I'll check out your posts.

Basically, unless CF is counting static asset network requests etc. For what it's worth, GA also miscategorizes some bots as humans as well.

(Insert spider man meme)

Good to know, thank you. Would you do this by fully blocking particular ASNs? Or something more granular?

Fair enough

That doesn't change the data/math brother

What's your strategy?

Not asking for trust, just sharing the data/math

Totally. I'm sure this campaign was inspired by sloppy vibe coding

This is a random sample of completely unrelated websites, which indicates that the total scale is much larger. This is not saying that it is difficult to make thousands of requests…

Looks like many of the paths relate to AI coding tools. There are some examples below the chart

Another interesting thing here is the paths they're targeting, many are for newish AI coding tools

Definitely, the point here though is there is a stat sig surge in the chart in the last week, across thousands of websites. At least a surge in this particular spoofing pattern.

Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.

I made this, let me know if you have questions or feedback.

Yup, here is an example with all listed agents, although you probably want to modify it to not fully block all of them: https://darkvisitors.com/robots-txt-builder

Yeah they respect robots.txt: https://commoncrawl.org/faq

Thank you, will try to confirm this.

You're not missing anything, they're just for any humans reading the file who want to know more about the agent. This is also just an example file.

Here it is: https://darkvisitors.com/robots-txt-builder

Right here: https://darkvisitors.com/robots-txt-builder This example blocks everything, which is probably not what you want, but it's meant to be a starting point.

It's updated daily. Thanks for posting by the way.

Ha thanks, it was a shower thought