Hi, murex author here. Murex is not a POSIX compliant shell but there are some lessons I've learned developing murex which relate to your points:
1. I do this in murex but actually the performance improvement barely registers in benchmarks because that's not where the real bottleneck is. It's forking and it's passing data via standard streams (stdout et al) rather than sharing memory. There's lots of expensive syscalls to do stuff that you wouldn't need to do in a monolithic service. It's also worth noting that shell scripting is designed to be a highly dynamic language - even command names can be variables(!) - so there is a limit to just how much up-front compilation and caching you do (at least if you want to retain POSIX compatibility).
2. No 2?
3. I also do this to some extent in murex too, except it's not coreutils but rather a set of additional utils specific for more complex data wrangling (JSON, YAML, TOML, CSV, etc). Unfortunately it comes with a set of additional complexities, any of which are deal breakers:
i) If you don't fork then processes don't register PIDs so you can't easily kill a long running process or command run in error. To get around this in murex I had to create a new, separate, function list. This list only includes processes launched from murex but it has both external files (awk et al) as well as builtins. This works for murex because murex isn't going for POSIX compatibility but this would never fly with Bash because you then need to train users to use different `kill`, `ps`, etc commands.
ii) Many users take advantage of job control, which is where you can stop, resume and background already running processes. This works via signals (like SIGINT which you'd be familiar with) and is controlled by UNIX/Linux and not the shell. To get this working with non-forked builtins takes a hell of a lot of hacking and, once again, breaks POSIX compatibility. In fact I ended up removing support for SIGSTSP from murex builtins because it wasn't reliable (though I do plan on adding it again at some point). I also have a bug where external commands don't get sent a SIGTTIN (signal to tell background processes to pause if they're reading from stdin) and that's largely due to some of the hacks I've made regarding your first point.
iii) re-implementing coreutils is non-trivial. In fact calling it "non-trivial" is a massively understatement. And that's without touching the other utils you've highlighted. This effort alone is enough to be a deal breaker without taking into account i and ii.
The reason busybox works is because it's a subset of coreutils. In fact busybox can be a little jarring at times because it is missing so many common flags. However it does just enough to be a useful emergency / embedded shell.
4. This is how murex approaches sub-shells but you're still limited by the caching problems (raised to point 1) plus the forking requirements (addressed in point 3) in terms of just how much additional performance it will squeeze. However out of all the points you have discussed, this is the easiest to implement.
It's also worth noting that fork-bombs (or equivalent exploits) will always be possible as long as your code has functions. Removing sub-shells wouldn't prevent someone from writing a self-referencing function.