Earlier quoted context omitted.
Heuristics are about optimizing between false positives and false negatives. Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents. (For some reason, the developers of these stealth systems think that variety beats commonality. Maybe it makes sense if they're specifically trying to overcome Apache mod_security's signatur…
> Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents I work in this space and we just use fingerprints we collect from actual users over the previous month. The hardest work is: 1) reverse-engineering the javascript of the CAPTCHA/fingerprinting solutions so that we can collect and encapsulate the fingerprints correct…
I suppose there are verticals where the data is so valuable, and the garden around it so walled-in, that you could build a whole IT business with a custom scraping stack just around extracting that data to then resell it. (I presume that's the business you're in.)
But for most verticals, the "attackers" you'll see in your logs aren't people building a data-broker business, and so aren't building their own secret-sauce anonymity from scratch; rather, they're end-users who want to do an end-run around your rate-limits, commit promotion fraud, etc., and so want to buy anonymity as a product, script-kiddie style. And "anonymity as a product", sold publicly (rather than through high-value contracts) tends to suck. It's script-kiddies buying from script-kiddies, with no real engineering in sight.
> I work in this space and we just use fingerprints we collect from actual users over the previous month.
Are you sure you're not in a citogenesis cycle? How sure are you that some of those "real users" aren't your peers' stealthed bots, who in turn picked up those fingerprints from unknowningly observing other stealthed bots in their logs, who...