But you do make mistakes. All humans do. Maybe you had an argument with a family member earlier in the day, and your mind wanders and you don't notice a red light. Maybe it's night, and raining, and some unexpected glare combined with debris in the road causes you to crash. Maybe you do drive tired, just once, even though generally you're strict about not doing so. Having a perfect driving record requires both luck (that no one else around you screws up) and constant vigilance on your part. Blemishing that record only takes the tiniest mistake, just once. No one, literally no one, is immune to these factors.
As much as I dislike the term "accident" when talking about car crashes (because, to me, the implication of the word is no one has to take responsibility), sometimes things just happen, because we are imperfect beings with imperfect nervous systems and imperfect perceptions and imperfect reaction times.
Self-driving cars will be better at a lot of things, but, yes, possibly worse at others. They have the potential to eliminate many causes of crashes, but might add a few new ones.
When you're on a plane, you're trusting not only the pilots, but a ton of complex avionics software. Why is that ok, while trusting self-driving isn't? I get that the two tasks are very different, and self-driving will require more sophisticated, nuanced software, but in both cases you're turning your safety over to a computer. That didn't work so well with the 737-MAX, but no one is talking about scrapping modern aviation because a bunch of people died due to bad software decisions.
The problem is the illusion of control. People think that driving their own car means they're in control of nearly every possible outcome, but in reality, they're not. Plenty of things can happen in a car that are out of the driver's control, even without another vehicle involved.
Another part of it is that people (Americans especially) can be excessively individualistic. Many people will balk at a solution that will result in (just making up numbers here) 25% fewer deaths if it means they personally will have a 0.001% greater chance of dying. Frankly, I find that mindset really worrying in a society, even if it can be understandable.
(Somewhat relatedly, I recall an episode of Star Trek TNG where one possible solution to the problem du jour was to give the computer full control over propulsion in order to save the ship. And we're talking about a futuristic computer that could probably flawlessly simultaneously self-drive every car currently on Earth without breaking a sweat. But in the end, blatantly pandering to our "human control is always superior" biases, the computer was found to be not good enough, and humans saved the day. Even more telling, I believe it was Captain Picard who took manual control; they didn't even have Data, the android, do it!)