There is a pattern that has been bothering me that I think feeds in to this but I haven't had the time to fully flesh out an essay on it yet.
We as a profession spend a lot of time _solving the same problems_. This isn't necessarily a bad thing, different implementations allow for specialization in unique but important ways. Where I think we've gone wrong is that we can no longer generically re-use a lot of code between code bases because a lot of those libraries are written in dead-end languages.
What I'm referring to as dead-end languages are any programming language where you can't use library code independently outside of its ecosystem. Golang, Erlang, Javascript, Python, Ruby, the entire Java land of languages are all one big ball of intertwined dead end ecosystems, even Rust to a lesser extent. Any library written in one of those languages is locked in to that ecosystem and will never have a chance at becoming a generic foundational building block for systems outside their ecosystem.
One of the reasons we're even able to rapidly build so many complex systems is the foundational libraries like libcurl that have "solved" a problem well enough and is reliable enough that it is effectively an easy default decision to use them. These are libraries that have more or less solved some hard problems sufficiently that other engineers can mental model them away without knowing the implementation or protocol details.
I've seen others compare these modern methods and tools to old-school in-house one-off development and how difficult that made things. This is the same effect but rather than lock in at a company level, its lock in at a language or library level (don't get me wrong this is generally better than random in house one-offs). If you're familiar with the Golang net/http package that mental model can't be transferred to another language and there is no way to expose that functionality to a language other than Golang due to how the language itself is designed.
As frustrating, old, decrepit, and unsuitable for a lot of things as the C ABI is, any language that can produce a library that exposes its functionality using the C ABI is table stakes right now to avoiding the sprawling landscape of language lock-in. Even in languages that support exporting libraries using the C ABI, there is always that concept of 'other' that seems so problematic to me. It's not _bad_ writing a library in Rust, but that boundary between Rust-land and the other is uniquely un-interoperable or overly repetitive in its own ways requiring layers of abstraction and special behavior to work. For example if you have two separate system libraries written in Rust that do all their work behind the scenes using tokio are they actually going to be sharing that runtime? No. There is no common libtokio.so file on the system, no cooperation or resource management between the two, and no common library to update if a security vulnerability gets detected (for the pedantic, I'm referring to pre-compiled distributed libraries as a system building block not the common source you can compile on your own). This specific problem of bundling specific versions into the compiled artifacts makes inconsistencies in the systems running the code have a lesser effect, but you end up having to deal with log4j like situations where you're entirely dependent on the packager, maintainer, or vendor to handle your security updates and trust that they got it right.
I think one of the big reasons we're experiencing this spiral of complexification comes from the fact that we're not generating those foundational building blocks any more. There is no refinement tuning out the complexity of the system and distilling best practices into library defaults. There is no common underpinnings being generated that can be maintained, understood, and diagnosed system-wide. We can't reason about this utterly shattered set of walled ecosystems.