I’ve had similar experiences, but I wonder how much of this is truly inevitable because software development is complicated and requirements often change along the way, and how much is simply bad management.
I’ve noticed that over time the prevailing culture in software development has drifted more and more towards “thinking small”. This has certainly been encouraged by the shift to web and mobile apps in an always-connected, always-updatable world.
First it was just the code. Short functions, few parameters, shallow nesting. Make everything testable and maintainable!
Then it was the commits. Release early, release often, merge WIP straight into master in your CI/CD system. We need fast feedback and short cycles to avoid unnecessary conflicts!
Then it was the whole process. Break everything down, do one small thing at once, move it across that Kanban board, next sprint please. We need focus and visible progress!
The danger with all of these is the same: they come from good intentions and even have an element of truth behind them, but they can also mean the big picture gets lost. There’s no coherent vision shared by everyone involved. No-one is watching all the extra dependencies that connect the many small parts. Tech debt compounds. Eventually we are forced to acknowledge that there are “challenges” but even then we somehow convince ourselves that those were inevitable, even though they weren’t really there before.
Yes, of course our early-merged WIP hidden behind a feature flag in CI is completely different to the feature branch we used to use. Now we know everything builds before it can be merged, even though nothing is really testing that all the different combinations of WIP actually work together, so that’s a big improvement on before. And since obviously we have a reliable, automated process for backing out any unfinished code that doesn’t make the cut later, we’re also much better off than the old situation where we’d just have sidelined that feature branch and never merged into the next level up in the first place!
In the real world, this leads to exactly the mess described in the parent comment. There’s no coherent vision for anything big that is shared by everyone involved, no considered and consistent structure for the software architecture or the APIs or the data formats. If we rely on emergent properties that evolve organically, we also run into evolutionary dead-ends, and what survives might work but also be messy.
Personally, I don’t really buy the argument that this is inevitable. Five players can make a great jazz band. With 100 players, you probably want an orchestra with a conductor and sheets and the occasional featured soloist. Neither jazz nor orchestration is “better” or “worse” in absolute terms but they are certainly different.