I don't know much about DDoS protection, but wouldn't it be possible to add token to UDP packets? So the client first connects to a TCP server and gets 4 bytes token unique per IP. Every UDP packed should start with that token. Since an attacker with spoofed IP can't get a valid token, their packets can be ignored with a cheap check. Might be even possible to use iptables/XDP for it?
We have a small team who work on this project during our free times and most of us just don’t currently have the time to dig into conntrack/nftables/xdp.