Earlier quoted context omitted.
I think this is different because I'm full on opening an instance of vim. (I didn't know about GNU Readline though and it looks awesome - I'll definitely try and work it into a project at some point).
I'm only using GNU readline for my bash prompt, but if I press v it spawns a full-on $EDITOR instance to edit the commandline.
Just so others know, you have to put bash in vi mode and press Esc before v (otherwise it's ctrl+x ctrl+e).
set -o vi