Am I the weird one? I usually have 3/4 terminals open at a time and rarely open new ones. Terminal startup speed is a non-issue for me. The only thing I demand to be fast on my terminal is grep reverse search (ctrl+r) and of course typing a character. But if your terminal can't keep up with your typing speed there is something deeply wrong with it.
I don't remember when I did it, but it looks like I must have gone through this at some point (maybe due to using GNU Make a lot? Or perhaps it was some other tool) - my zsh setup does a bunch of autocomplete setup only in the interactive case, and it seems to help a bit with startup time, at least on macOS:
% for i in {1..5}; do /usr/bin/time zsh -i -c exit; done # zsh in interactive mode
0.05 real 0.03 user 0.02 sys
0.02 real 0.01 user 0.01 sys
0.02 real 0.01 user 0.00 sys
0.02 real 0.01 user 0.00 sys
0.02 real 0.01 user 0.00 sys
% for i in {1..5}; do /usr/bin/time zsh -c exit; done # zsh in non-interactive mode
0.01 real 0.00 user 0.01 sys
0.01 real 0.00 user 0.00 sys
0.00 real 0.00 user 0.00 sys
0.00 real 0.00 user 0.00 sys
0.00 real 0.00 user 0.00 sys
For the interactive case, I don't really mind (within limits - the worst case, on macOS after a reboot, takes several seconds, and that's tedious). I also start new interactive terminal sessions fairly rarely.