Website unresponsive: diagnostic steps and blocking the AI crawlers

Blocking IP ranges is just whack-a-mole though right? Glad you got the site back up but this necessitates a high level of vigilance going forward.

Can a softer throttling approach work? Rate limit the naughty IPs when they go over a threshold, or even increase rate limits as CPU/RAM usage climbs?