Do people still use Vagrant? Last time I tried Docker on Mac, it was painfully slow. So I kept using Vagrant, which used more disc space, but very fast, and the Intel Macbook pro fan never kicked in once like Docker did.
As far as Vagrant being faster than Docker, it's possible the qemu backend it uses was hardware accelerated and for whatever reason the Docker VM wasn't? Or perhaps you were using Docker Desktop which does take up a lot of compute capacity away? Try using the docker cli without docker desktop and see if it works better for you.