Earlier quoted context omitted.
In every single one of these cases, it begs the question whether "JS" is the problem or "backwards compatibility". There are a huge number of inefficiencies that can be fixed with a full rewrite. There was a recent conversation over ViteJS (a pure JS bundler) vs rust-based tooling, and when you dig into the numbers the real difference is SWC vs Babel. It raises the question whether a new transpiler written in JS can…
SWC is written in rust, and Babel is written in javascript, so you've proven the OP's point. Vite can be configured to use SWC. The recent conversation was about Vercel benchmarking a tuned Turbopack+SWC vs. a default Vite+Babel, not really an apples to apples comparison. When Vite is configured to use SWC as a compiler, Vite's HMR gets faster; but it's not the default for compatibility reasons.
And yes, Babel can also be made faster if enough effort is dedicated into it. it's not an impossible feat.