Earlier quoted context omitted.
I'm an open book! Ask any question that you would like! Here's the thing - all of this info is out there (largely in other HN threads on this topic) and I'm nothing special in my field-of-knowledge ;) I'm confident if people fully prevented my "abuse" they'd start to block actual users... simple as that.
What's your take on ML for bot classification? How successful has that family of strategies been in your opinion? One could speculate on what particular features of client behavior a model would hone in on to detect a bot, but it would actually likely be unexpected behavioral oddities not shared by legit clients that a human developer's intuition wouldn't think of.
When it comes to a web "interaction" there's a ton of differences depending on the job at-hand. If it's jumping on a Nike drop you're not going to run into those sort of methods where-as if it's scraping hundreds of thousands of SKUs there's a huge chance you will.
When a human opens a "conversation" with a website usually it's for a specific action which is a small burst of data, ie: looking at a few products on Amazon. So as long as you can design your bot's "conversation" with said website in the same way you can totally bypass ML detection (and everything else for that matter).
Basically if you think of ALL of the unique variables that are happening back and forth in that conversation it's a finite list. If you can check off each and every one of those you can get past anti-bot measures no-problemo, it's just a matter of budget/time. ML can HELP to identify bot traffic, but if you're sending perfect headers/traffic/browser metrics/CC numbers/shipping addresses (this list gets long) you're still going to squeak by with an acceptable risk score without a problem. The other thing is going raw-HTTP request often bypasses almost all of that crap, way more than one would think! I will often get "fingerprinted" with a very valid browser, then continue my work after treating their site as an API; ie: talking directly to the backend (or rendered HTML page) and no one else.
The #1 thing I'd say where bot classification works is network reputation... so I use services that allow me to proxy-out to VERY reputable networks that are actually business/residential ISP connections. This lets me get past the majority of countermeasures because if they start blocking those sort of IP addresses/ranges they're going to block real users at some point. Unfortunately, highly reputable proxy services do cost money - but for someone who does this professionally it's all about budgeting etc.
To implement these systems well takes a really keen implementation between your front-end and back-end and most companies (even the big ones) don't have the development sophistication to pull it off. Those that do are just more expensive because you'll be bouncing around between 40 different "origins" with unique browser profiles.
Sorry for the essay =)