I find the language/ecosystem-specific idioms really interesting from a historical perspective. I feel like it makes for great reading if one wants to know the common problems involved in building packages for a given framework. I spent some time fixing some packages on Darwin for the most recent release, and I was surprised by how many of those were stdenv-based. I don't really write C/C++, so I don't end up using the stdenv builders myself.
At some point, one of my fixes used an "old" idiom; it was valid, but it had fallen out of use. It was just the first example I found for what I thought would fix the package, and it turned out to be correct, but the new idiom was cleaner and clearer. Nevertheless, they took my fix with the old idiom, so it will continue to propagate for what is likely the same reason. Some of this is just a result of the size and scope of the work going on, especially around release regression time.
In a separate personal example, I went down a very deep rabbit hole of trying to figure out why I couldn't use a shell as a base for a derivation that was essentially just packaged code used as a payload for an external system. It turns out that shell derivations are special things and can't be used directly for standard derivations. My eventual solution, using them indirectly, seemed hinky and didn't incrementally cache well, but it worked.
I agree very much with the problems you've described, but I am a bit more bullish on the potential of the larger ecosystem to outgrow at least some of those problems. I think flakes are the key to enabling better decoupling from nixpkgs. One could rewrite a saner stdlib as a flake, which could then get pulled back into nixpkgs as a flake input. There are promising tools like devshell and flox that are making a serious attempt at unlocking broader adoption through ease of use.