Even with Agile and Scrum waterfall will sneak in

Indeed. They way I understand Agile/Scrum in enterprises is basically a dogmatic process around implementing a bog standard CRUD app Nth time. Minor details keep changing from customer to customer.

It should be clear to folks that complex technology products in past were not developed by caricature of Waterfall method that Agile peddlers show on slides. Neither today interesting engineering products are developed following some "methodology industry".

I have about one third of the experience you have, but still I ended up in a place that was doing something very close to the mythical waterfall, and it was a horrible system.

We started with a one year long release plan, with a defined set of features that were supposed to make it in. A few weeks were spent defining functional specs for the features, negotiating them with PMs, and finally handing them over to the QA department (which was entirely separate). Then, dev work would start on all these features, while QA would come up with a series of Acceptance tests, Feature Tests, System Tests, Final Tests, Automation test plans etc. - all based on the FS.

Timelines and Gantt charts would be produced, deciding how to fit the features and required testing into the 1 year timeline.

After many months of dev work, the first builds would start making their way to the QA department, who would start tentatively running the Acceptance tests for the features that were supposed to be delivered. Some rapid iteration on any bugs affecting the Acceptance tests would happen here. Once the Acceptance test pass rate was high enough for the feature, QA would start working on Feature Testing, logging Bugs, while the dev teams would move on to other features. Occasionally QA or other devs would complain about blocking bugs that would have to be addressed ASAP, but otherwise bugs would pile up.

This would continue for all features, with dev teams keeping an eye on the total bug counts trying to make sure they would stay below some limits that had been set in a Bug plan (typically this meant keeping below 2-300 bugs for a team of ~5-6 people).

Finally, all features would be delivered to QA and passed acceptance, reaching the Feature Complete milestone. At this time, the dev team would start work in earnest on bugs, while the QA teams would continue with Feature Testing, logging new bugs.

This would continue for 1-3 months typically, until the Feature Testing would reach a decent pass rate, reaching the Feature Test Complete milestone. QA would now start on Performance testing and Early Final Testing. Often this would be the first time multiple features would truly start being used together. Dev teams would still be on full bug fixing mode.

When bug counts were low enough and Early Final Testing had a good pass rate, the product would reach the Code Chill stage - only important and safe bugs would be allowed to be fixed anymore. At this time, Final Testing would start in earnest, with the most complex cross-functional tests. The dev team would be finalizing the remaining bugs, until Code Freeze was reached - the vast majority of testing done, and only critical issues being allowed to be considered for fixing.

Of course, during this while, deviations from the schedule devised initially would be monitored constantly, with pressure coming not only from upper management, but also between QA and dev, as delays on the dev side would easily translate to delays on the QA side.

Customer feedback on previously released versions of the product, or new opportunities, would be very hard to squeeze in, causing panic and chaos in all the well laid plans. More often than not, it would be left to next year's release, meaning the average time between an opportunity and a release containing a feature to address that opportunity would be ~1.5 years.

Dev and QA would often be at odds, blaming each other for schedule slips, complaining about silly bugs, trying to pass the buck.

Of course, overall the whole thing worked, but slowly moving to a more Agile working model was a massive relief to everyone and created much better team cohesion, much less pressure on deadlines and deferred features (in the past, a feature getting kicked out of the current release meant it would only come one entire year later at best), and a much better product. Huge issues in design would no longer be treated as bugs, discovered weeks after the initial delivery of a feature. Tests would be designed by dev and QA together, often being run iteratively days after the code is written, leading to much better quality and corner cases getting caught early, not late. Process adjustments are now local team decisions, whereas before they required VP level involvement and agreement and coordination between all teams.

Fully agree, agile is fine.

The real problem is that Scrum is not agile at all: Scrum is a fat "process" that enforces "following a plan" (regular, rigid meeting structure), "creating comprehensive documentation" (user stories, specs, mocks, task board) and "contract negotiation" (estimation meetings, planning poker). In that way, it's the exact opposite of the original agile Manifesto: http://agilemanifesto.org/

The only thing Scrum has ever been great at (and why it's continually chosen despite proven harmful to development productivity) is that it's great at simulating continuous progress and control to management. Other than that, it's simply fostering architectural mess and a political, dogmatic attitude from and towards everyone.

It all starts with a clear product vision from the top and defining how succeeding on it looks like.

If you don't know where to go, Scrum will get you there faster and in even more directions at once.

Or, to cite one of my all time favorites:

"Building intuition on how to make good decisions and cultivating a great relationship with your team will get you 95% of the way there. The plethora of conceptual frameworks for organizing engineering teams won’t make much difference. They make good managers slightly better and bad managers slightly worse."

— https://www.defmacro.org/2014/10/03/engman.html

I am about as old as you. Unless you started at 13. The problem was that projects spent months doing analysis and design and never ship everything. Also agile recognizes that you don have all the answers at the beginning; you need to experiment and see how things should work. There was also a tendency to treat people as factory workers that you could assign any work and they just convert specs into code. We still do that with Scrum. :(

If I read the paper correctly, the iterative approach considered as wasteful, and two-stage waterfall is recommended instead: documentation and planing at first stage, to find weak points first, then execution of the plan. IMHO, it was designed for times when computer time was expensive.

I did a few projects in waterfall style about 20+ years ago, but my memories are faint. Our PM used a UML modeling tool to model classes and relationships, then exported it into Java. We wrote a lot of useless documentation. I developed a good habit to write documentation immediately, when memory is fresh, to avoid the pain of writing it later.