Earlier quoted context omitted.
> Cars don't randomly accelerate at 100,000m/s/s in some direction they aren't pointed What if they are hit by a truck? Maybe not 100,000 m/s^2 but if you assume that cars can't accelerate in directions they aren't pointed, you will be wrong at the worst possible time.
That's why I elaborate, and why I chose that number. The only way something actually accelerates like that is an error, or it's an error.
A better approach would be to include temporal data in the inputs to the neural net so it can learn how to do the prediction and filtering itself using all the context available in the input imagery, instead of processing each frame completely independently and feeding low-dimensional symbolic results into some other system. But you'd need a very large dataset and a very large neural net.