My Debian VM starts in seconds. Last docker image I had to use was Ubuntu, and it was enormous. For local purposes, I didn't really see the point to using a docker container. That, and I don't particularly trust overlay filesystems.
The official base Ubuntu image is really not particularly large, I'm assuming you were using an image with additional things installed? But regardless, once pulled, it is cached locally. Be sure to reference the image by specific tag/sha rather than latest to ensure you're not doing excessive pulls. Overlayfs 1 had many issues. Overlay 2 was buggy for a long time and we needed to patch in aufs in production for stabi…
I think you underestimate the issues with that. Why do you think VM companies haven't jumped on this bandwagon?
The closest thing I see to that are the restricted micro-VMs like Firecracker. And a big part of the reason they exist is to support the needs of containers.