This is why I still see it as a novelty. In time this will most likely change. Currently, my issues with it all boil down to trust and efficiency. Let's use two simple examples. Uber and movie ticket booking.
Let's say I wish to book an Uber. Usually, I open pop open the app and hit the request button. Doesn't take long at all. I get visual confirmation through the map and the little lollipop Uber driver photo. It's fast and I have the feedback to know that what I set out to do has been done. What if I had used the assistant? Well, now I have to waste a lot of time chatting to it. Sometimes it reads my address wrong or it doesn't understand Australian. So after a few mishaps, I finally get told that it's on the way. Okay. Then the twitching starts. The need to check that it was actually entered properly kicks in. So I open up the app to check the lollipop heating-seeking Uber driver is on target. I end up at the same place I attempted to avoid. Yet this avoidance was less efficient. I didn't receive enough feedback so I was unable to trust.
The second example is ordering a movie ticket to go see Batman 27 with my wife. I ask the assistant awkwardly to book for me and give it a time. It tells me that it's all good and dandy. The trust issues then kick in. So I go head off to my e-mail to triple check my confirmation. I wasn't active during the booking process, so the feedback is all I have to work with. It's date night too, so I can't waste it or I'll be sleeping on the couch. I've just wasted time.
Then it occurs to you that by bypassing their system, you may have possibly missed some little announcement or some benefit that is only visible to you during the booking process. For example, doing it manually may show me that there's a time slot that has more ticket availability than the one we were originally headed to. This means an emptier cinema so that I don't get weird looks when she starts telling me about her day mid-movie. The assistant wouldn't have told me that. The assistant also can't toggle around the tick boxes and drop boxes that would have told me I would have been able to do a 2-for-1 deal if we had gone an hour later.
Being there during the actual process isn't always a bad thing. Abstracting away these manual processes would be fine if all processes were identical. They're not though. You're losing too much control and being provided with too little feedback. Now, I can very much see the benefit of the assistant while driving. It makes sense there. You're usually unable to hold the device and that form of interaction is a best fit.
I don't think it's ready yet. I can do everything faster manually. It doesn't add anything at the moment, it just subtracts.
It's the same reason I have no use for Cortana on Windows 10 or Siri. Why would I talk to the computer when it's considerably more efficient to just start typing.