Here's another guess: a DDoS to steal your attention and mask other intrusion attempts.
> Defenses need to have broader rate limits across more than just IPs (ASNs, hostnames, etc.).
The description/approach seems static and limited? Why not maintain a leaky bucket that counts each request (tickets/points) with higher cost for expensive requests (404, redirects). As the IP's reputation deteriorates (IPv4/32), it begins to spill over to a broader subnet like IPv4/31 then /30 and so on. Fight adaptivity with adaptivity. // maybe I describe something totally obvious, I'm not involved in the web ddos protection side of things.
> Attackers actively search for non-cacheable paths (e.g. dynamic redirects, search endpoints, and 404s).
To continue with my previous point. As long as individual server's resources permit (memory, socket limit) stall requests before processing them. Low reputation IPs get stalled for longer and the requests that exceed the queue get dropped. The idea is graceful degradation:
A good rep IP will not be stalled by sleep(). A poor rep IP will be stalled, but eventually receive its answer instead of some 429/403 (i.e. a user who opened many tabs at once). A bad rep IP will be slowed down by the wait times + rate-limits (queue exceeded) before getting completely banned for good. The point is to have more granularity before throwing errors at random users at the server-level.