Another obvious problem with having the neural network map from vision to steering is that you can't make it take decisions based on what it hasn't seen yet, but will see. E.g. changing lanes because to reach your destination you'll have to make a right turn. The authors note that E2E learning makes for a better smaller system, on account of being free from human-imposed concepts. That's fine if that's your only goal…
This doesn't describe the (very basic prototype) system in the paper, but there is no theoretical reason there cannot be a recurrent net that plans ahead (obviously developing such a system is extremely difficult)
What would you give it as an input, though? Real-time screen captures of Google Maps in navigation mode?