I find it odd the way people talk about moving through the levels sequentially without really acknowledging there's a massive phase change that happens somewhere between level two and four, where a self-driving car could easily be way more dangerous than the average human driver. That comes as soon as you have a situation where the automation has to be able to handle all foreseeable scenarios without failing because…
Just remember that self driving cars do not have to be perfect, they just need to be better than humans. The use is at just over 1 death per million miles. If self driving cars can reach .5 they are better already. If the cars can detect the times they are bad and pull over (screw your appointment) that might be enough.
But...
It isn't just deaths that need to be prevented. It is a combination of total damages, delays, deaths etc. If self driving cars take forever to make a turn or have other AI failures like driving too slow, it won't be acceptable to a large number of people.
In addition, you will see a situation in the future where partially automated cars Level 2 or 3 are safer than current driving modes. That means that in order to go from level 3 to level 4, you need the car to be safer than level 3.
I want a fully automated self driving car so badly, but I suspect I have a long way before I can hop in a car with no steering wheel and sip an IPA while being rushed home from a tough day at the office.