I think the personal freedom and privacy implications of self driving cars with centralized communication are rarely talked about. What happens when your vehicle decides you can't travel to an area of civil unrest (for your safety), or refuses to drive you to the gun store?
For one, areas of unrest are already restricted by physical police and national guard barriers. For example, during the unrest in 2020, Chicago physically raised its drawbridges. Anyone who really wants to get somewhere difficult would probably choose a bicycle or legs, so you need physical police barriers in that case.
Refusing to drive you to the gun store sounds kind of silly when the government would more easily just ban gun stores (the right to bear arms doesn't include the right to buy them at a retail store). I realize you're just giving out an example but it was not a great one. Preventing you from driving somewhere is a simple as putting up a gate and closing it.
Also, driving at all is not a right. It's a licensed privilege that can be taken away if you break traffic laws or drive while intoxicated.
I think your perspective is coming from a car-focused area of the country (which is most of them). But, someone in NYC or Chicago might point out to you that they use a transit card that has the ability to track their movements through stations, and that they mainly travel by buses and trains that have government-owned surveillance cameras all over them. Despite this, it has has failed to turn into any kind of dystopian urban nightmare.
What I'm getting at is that the technology is blamed for things that are actually just public policy, laws, and law enforcement. The privacy implications for using a smartphone in the Western world are a lot different than in China, even though the technology itself is identical.
What we actually need is legislation surrounding automotive communication and centralized control and privacy. The rights that automobile owners have should be better enshrined in law rather than trusting auto manufacturers. This, I think, will eventually happen. We're still sorely lacking a general nationwide privacy law on the Internet in the US.