>> If you ignore a dependency and try to fix it later, it will be more expensive. More time, more effort, more thinking. And it will require the same level of coordination that you tried to avoid initially.
Systems Thinking
91–100 of 123 posts
Re: Systems Thinking
#92> you lay out a huge specification that would fully work through all of the complexity in advance, then build it. This has never happened and never will. You simply are not omniscient. Even if you're smart enough to figure everything out the requirements will change underneath you. But I do still think there's a lot of value into coming up with a good plan before jumping in. A lot of software people like to jump in a…
> A lot of software people like to jump in and I see them portray the planning people as trying to figure everything out first. My approach, especially for a project with a lot of unknowns, is usually to jump in right away and try to build a prototype. Then iterate a few times. If it's a small enough thing, a few iterations is enough to have a good result. If it's something bigger, this is the point where it's worth…
I've seen some issue with this approach is that management will want to sell the prototype, bypassing the "rewrite from the lesson learned" step, and then every shortcut took into the prototype will bite you, a lot..
And things like "race conditions"/lack of scalability due to improper threading architecture aren't especially easy to fix(!)..
Re: Systems Thinking
#93Waterfall specifications have never worked as advertised.
The 1980s and 90s were full of DOD-497 multi-kilogram documents being analyzed atomically to determine the specification, and they rarely came in close any of the 3 main dimensions of success: time, quality, or cost.
On the other hand, neither has Agile with a capital A, with the ceremony of documents replaced with the ceremony of JIRA tickets and t-shirts.
Re: Systems Thinking
#94> you lay out a huge specification that would fully work through all of the complexity in advance, then build it. This has never happened and never will. You simply are not omniscient. Even if you're smart enough to figure everything out the requirements will change underneath you. But I do still think there's a lot of value into coming up with a good plan before jumping in. A lot of software people like to jump in a…
I'd like you to go look at PRINCE2 and SSADM. Or read the original Royce paper - https://www.praxisframework.org/files/royce1970.pdf was written explicitly to term this Antipattern "Waterfall." (Note that Royce marks it as an antipattern.)
I discussed some of this in https://www.ebiester.com/agile/2023/04/22/what-agile-alterna... and it gives a little bit of history of the methods.
We are nearly 70 years into this discussion at this point. I'm sure Grace Hopper and John Mauchly were having discussions about this around UNIVAC programs.
Re: Systems Thinking
#95Earlier quoted context omitted.
> A lot of software people like to jump in and I see them portray the planning people as trying to figure everything out first. My approach, especially for a project with a lot of unknowns, is usually to jump in right away and try to build a prototype. Then iterate a few times. If it's a small enough thing, a few iterations is enough to have a good result. If it's something bigger, this is the point where it's worth…
I've seen some issue with this approach is that management will want to sell the prototype, bypassing the "rewrite from the lesson learned" step, and then every shortcut took into the prototype will bite you, a lot .. And things like "race conditions"/lack of scalability due to improper threading architecture aren't especially easy to fix(!)..
You don't ever make the prototype public.
Also, there's a certain point where you can't avoid management sabotaging things.
Re: Systems Thinking
#961. This sounds great in theory. In theory there is no difference between theory and practice, but in practice there is.
2. I would be more receptive to this argument if they had listed some famous examples of successful, large systems that were built like this. On the other hand, I can easily list many failures: FAA Advanced Automation System (1980s), IRS Tax Systems Modernization (1990s), UK NHS National Programme for IT (2000s).
3. Waterfall vs. agile is a continuum. Nobody plans everything, down to each if-statement, and nobody wings it without some kind of planned architecture (even if just inside one person's head). Where you are on the continuum depends on the nature of the problem (are all requirements known?), the nature of the team (have they done this before?), and the criteria for success (are there lives depending on this?).
4. The analogy to building a building is flawed. At large enough scale, software is like a city, and all successful cities have gradually evolved in complexity. Come back to me when someone builds a 1-million person arcology on some island in the Pacific.
5. Just as some PhDs are sensitive about being called "Doctor", some software engineers are sensitive about being "real engineers". Stop thinking about that. What we do as software engineers is immensely valuable and literally changing the world (usually, but not always, for the better). Let's stop worrying about whether or not what we do is "engineering" and focus on what we do best: building complex systems that have never before existed on earth.
Re: Systems Thinking
#97“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.” Gall’s Law
Some systems require a total commitment to the complexity because it is intrinsic. There is no "simple" form that also works, even if poorly. In many contexts, "systems thinking" is explicitly about the design of systems that are not reducible to simpler subsystems, which does come up in many types of engineering. Sometimes you have to eat the whole elephant.
There is a related phenomenon in some types of software where the cost of building an operational prototype asymptotically converges on the cost of just writing the production code. (This is always a fun one to explain to management that think building a prototype massively reduces delivery risk.)
Re: Systems Thinking
#98> There are two main schools of thought in software development about how to build really big, complicated stuff.
> The most prevalent one, these days, is that you gradually evolve the complexity over time. You start small and keep adding to it.
> The other school is that you lay out a huge specification that would fully work through all of the complexity in advance, then build it.
I doubt many people (if they stop to think about it) actually fit into either of these two schools of thought, they're both extremist positions. It's like claiming that the US population is filled with people believing in either big government autocracy and communism or small government libertarianism and free markets. That's an absurd position to take, just as these opening paragraphs are absurd.
The most generous interpretation is that the author is setting up a strawman.
These are extremist positions. No one except a fool would think that they can design a truly large system from scratch before ever writing a piece of code, and no one but a fool thinks they can write the code for a large system without ever thinking about the design.
The reality is that it sits in between, the author figures this out by the end, fortunately, but wants something they can't have:
> There should be some balanced path in the middle somewhere, but I haven’t stumbled across a formal version of it after all these decades.
First, they at least admit that they're stumbling. That's good, groping around in the dark is not an effective way to find an answer, turn on the lights. You aren't the only one thinking about this subject.
Second, for small projects and simple projects, or rehashes of projects you've done before, the approach often doesn't matter. This only matters for large, complex, and/or novel projects.
Software development is a design process.
Again, in case this was missed: Software development is a design process.
The idea of separating design from development is foolish (BDUF). The idea of separating development from design is equally foolish (extreme take on Agile, see Ron Jeffries failing at sudoku because he tries to use a development technique, TDD, without thinking about the design).
Take the "over 3000 active systems" from paragraph 4. There is no way anyone could have designed all 3000 active systems (either as the 3k systems or in a compressed form) from scratch in a reasonable amount of time. The only reason the author can think of a better design is because they have a design, even if it's not formally documented. The existing software is the design [0] that they can draw from to come up with the better design.
But wait, the foolish BDUF people would not try to refine the system, they'd try and build a new system from scratch. Don't be a fool.
The foolish extreme Agile people would not look at the whole (or a large enough section) and think about refining it, they'd just add to it or change the existing systems.
The sensible person says, "Wait, I acn'
Re: Systems Thinking
#99> you lay out a huge specification that would fully work through all of the complexity in advance, then build it. This has never happened and never will. You simply are not omniscient. Even if you're smart enough to figure everything out the requirements will change underneath you. But I do still think there's a lot of value into coming up with a good plan before jumping in. A lot of software people like to jump in a…
> This has never happened and never will. You simply are not omniscient. Even if you're smart enough to figure everything out the requirements will change underneath you.
My best project to date was a largely waterfall one - there was somewhere around 50-60 pages of A4 specs, a lot of which I helped the clients engineer. As with all plans, a lot of it changed during implementation, actually I figured out a way of implementing the same functionality, but automating it to a degree where about 15 of those could be cut out.
Furthermore, it was immensely useful because by the time I actually started writing code, most of the questions that needed answers and would alter how it should be developed had already come up and could be resolved, in addition to me already knowing about some edge cases (at least when it came to how the domain translates into technology) and how the overall thing should work and look.
Contrast that to some cases where you're just asked to join a project and help out and you jump into the middle of ongoing development, not going that much about any given system or the various things that the team has been focusing on in the past few weeks or months.
> It’s not hard to see that if they had a few really big systems, then a great number of their problems would disappear. The inconsistencies between data, security, operations, quality, and access were huge across all of those disconnected projects. Some systems were up-to-date, some were ancient. Some worked well, some were barely functional. With way fewer systems, a lot of these self-inflicted problems would just go away.
Also this reminds me of https://calpaterson.com/bank-python.html
In particular, this bit:
> Barbara has multiple "rings", or namespaces, but the default ring is more or less a single, global, object database for the entire bank. From the default ring you can pull out trade data, instrument data (as above), market data and so on. A huge fraction, the majority, of data used day-to-day comes out of Barbara.
> Applications also commonly store their internal state in Barbara - writing dataclasses straight in and out with only very simple locking and transactions (if any). There is no filesystem available to Minerva scripts and the little bits of data that scripts pick up has to be put into Barbara.
I know that we might normally think that fewer systems might mean something along the lines of fewer microservices and more monoliths, but it was so very interesting to read about a case of it being taken to the max - "Oh yeah, this system is our distributed database, file storage, source code manager, CI/CD environment, as well as web server. Oh, and there's also a proprietary IDE."
But no matter the project or system, I think being able to fit all of it in your head (at least on a conceptual level) is immensely helpful, the same way how having a more complete plan ahead of time can be helpful with a wide variety of assumptions vs "we'll decide in the next sprint".
Re: Systems Thinking
#100I disagree with most of this article, but this part stood out: > the size of the iterations matters, a whole lot. If they are tiny, it is because you are blindly stumbling forward. If you are not blindly stumbling forward, they should be longer, as it is more effective. You are not blindly stumbling forward, you're moving from (working software + tiny change) to (working software including change). And repeat. If the…
exactly - and this is what good agile attempts to address - implement (importantly) user-facing function in exactly the smallest size possible (also important) with those two things you can build, identify issues / questions / problems in the fastest feedback loop as possible indeed, i would argue 'big iterations' are the ones where all the problems which the author mentions crop up in the first place!
> big iterations' are the ones where all the problems which the author mentions crop up in the first place!
That’s certainly my experience