Perhaps it is lost in translation; I'm not saying the fact that someone is driving drunk only matters when an accident happens, I'm saying that right until the moment a driver decides to get drunk, the law doesn't do anything. If at the beginning of the day someone decides to start drinking and when they are drunk they get in to a car and start driving, that's when the violation occurs. Not before that like PreCrime would.
The same can't apply to FSD because it isn't consistently 'driving drunk'. That analogy doesn't hold because it is not fixed software like a GPS-based navigation aid would be. Just like humans it does have more or less fixed parameters like the amount of arms and legs you have, that doesn't tend to change depending on your intoxication.
One could make the argument that it's not as much the "haha it is just like a drunk driver zig-zagging", but the uncertainty about the reliability. If a car with some autonomous driving aid drives across an intersections just fine 99 times out of a 100, and that one time it doesn't, that doesn't mean the car software was dunk 100% of the time.
Why FSD is not held to some standard, I don't know. I suppose that depends on how it is defined by local law and how the country it is in allows or disallows its use.
The problem with prevention and detection here is that like humans, the system is not in a static state. The trained neural network might be largely the same with every release, but the context in which it operates isn't, unless the world around it stops completely in which case two trips can be identical and because the input is identical the output can also be identical. Humans do the same, even if well-rested and completely attentive, knee-jerk reactions happen.
Holding FSD to a standard of a drunk driver isn't a valid comparison due to the non-static nature of the state it is in. This isn't even FSD-specific, even lane guidance/keeping assistance and adaptive cruise control isn't static, and those are based on pretty static algorithms. Even the PID-loops used on those will deliver different results on seemingly similar scenarios.
Perhaps we should stop comparing technology to humans since they are simply not the same. The static kind isn't and neither is a NN-based one. We can still explore results or outcomes because those are the ones that have real impact. And let's not fool ourselves, humans are far less reliable in pretty much every man-machine combination. But in human-to-human contexts we factor in those unreliabilities, and with machine-to-human or machine-to-machine we seemingly don't, which is pretty much the same problem you're describing.
This will be an interesting field of development, and if we simply take death toll into account, keep in mind that for some reason seatbelts were thought to have 'two sides of the story' as well when they were first introduced and later required. As with car seats for children and infants, a good idea might start out one way and over time (with the accompanying bodycount) it gets shaped into whatever we expect of it today. Same goes for aerospace, boats and trains, and that's even without taking software into account.