It seems like a mistake to remove manual controls entirely, not necessarily because of safety concerns, but because there are a fair number of times when you need the car to go somewhere without having a route. I'm thinking of things like: Moving your car ten feet so somebody else can get out of the driveway; handling double parking during alternate side parking hours; moving the car because you were waiting to pick…
I'm not sure why you and a few others think a joystick will be problematic. Of course, there will be a bunch of tasks/use cases for which these cars won't be suitable, but so what? there's many use cases for which bicycles and motorcycles are unsuitable, but lots of people still use them.
I can certainly see using one of these things for local get-around driving, because I hate to drive and my wife can't.