Let's assume your worst-case scenario comes to pass, and politicians decide (or are influenced) to outlaw manually-driven vehicles from public roads.
First, you would still be able to drive on private tracks; plenty of car enthusiasts already do this today. If your enjoyment is derived from driving fast, or taking curves, this should be able to satisfy your requirements.
Second, if your satisfaction is derived from aimlessly driving, simply enjoying the scenery and the zen of the road, this is not something that is incompatible with self-driving cars. Hell, there could be a "Scenic drive" button that randomly takes you on different routes every time.
If you are concerned that you will no longer be able to combine these activities (driving aggressively on long, rambling courses), I think this is something you are likely to lose out on. On the other hand, if this is the case, is it even appropriate for you to be doing this on public roads today?
If your argument is that you don't want to invest the time or money to use tracks or private courses for this purpose, then I think you should reexamine just how much enjoyment you derive from driving, and where it comes from specifically.
Ultimately, it goes to a core discussion of whether roads are for transportation or for recreation; I imagine similar discussions have occurred in the past between wagons and horseback riders, for example, or later, between horse-drawn vehicles and cars. Today, I would argue that more than 50% (and probably much more) of the cars on the road are being used for transportation rather than recreation, which suggests that modifications that streamline that use will eventually crowd out the remaining percentage through the tyranny of the majority.