Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systems Thinking

theprogrammersparadox.blogspot.com

Post reply on HN
Print view View original

81–90 of 123 posts

Re: Systems Thinking

#81

Post by wduquette »

Grady Booch said that any large system that works is invariably found to have evolved from a smaller system that worked. I've seen this cited as Gall's Law, from John Gall's 2012 book Systemantics, but I read it in a book by Booch back in the late 80's/early 90's. At that time the "waterfall model" was the conventional wisdom: to the extent possible, gather all the requirements, then do all the design, then do all the coding, then do all the testing, doing the minimum of rework at each step.

It didn't work, even for the "large" systems of that time: and Booch had worked on more than a few. The kind of "system" the OP is describing is vastly larger, and vastly more complex. Even if you could successfully apply the waterfall model to a system built over two or three years, you certainly can't for a system of systems built over 50 years: the needs of the enterprise are evolving, the software environment is evolving, the hardware platform is evolving.

What you can do, if you're willing to pay for it, is ruthlessly attack technical debt across your system of systems as a disciplined, on-going activity. Good luck with that.

Re: Systems Thinking

#82

Post by bluGill »

pjmlp wrote: post #51
godelski wrote: post #47

> you lay out a huge specification that would fully work through all of the complexity in advance, then build it. This has never happened and never will. You simply are not omniscient. Even if you're smart enough to figure everything out the requirements will change underneath you. But I do still think there's a lot of value into coming up with a good plan before jumping in. A lot of software people like to jump in a…

Yes it did, however it never works in pratice when it comes to integration testing two years later after the 2000 pages specification document was written, and passed down from the architects to the devs.

2000 page specification documents are rarely useful (if ever?).

You need smaller documents - this is the core technology we are using. This is how one subsystem is designed - often this should be on a whiteboard because once you get into the implementation details you need to change the plan, but the planning was useful. This is how to use core parts of the system so new comers can start working quick.

You need disciple to accept that sometimes libfoo is the best way to solve a problem in isolation, but since libbar is used elsewhere and can solve the problem your local problem will use libbar despite making your local problem uglier. Have a small set of core technologies that everyone knows and uses is sometimes more valuable than using the best tool for the job - but only sometimes.

Re: Systems Thinking

#83

Post by bluGill »

nasretdinov wrote: post #30
bestham wrote: post #3

“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.” Gall’s Law

I think the important part here is "from scratch". Typically when you're designing a new (second, third, whatever) system to replace the old one you actually take the good and the bad parts of the previous design into account, so it's no longer from scratch. That's what allows it to succeed (at least in my experience it usually did).

These days software has been done a lot. You should be able to find others who have done similar things and learn lessons from them. Considering microservices - there are lots of people who have done them and can tell you what worked well and what didn't. Considering using QT - lots of others have and can give you ideas. Considering writing your own framework - there are lots of others: look at what they do good and bad.

If you are doing a CRUD web app for a local small business - there are thousands of examples. If you are writing control software for a space station - you may not have access to code from NASA/Russia/China but you can at least look at generic software that does the things you need and learn some lessons.

Re: Systems Thinking

#84

Post by bluGill »

saulpw wrote: post #52
jackblemming wrote: post #27

Earlier quoted context omitted.

People misinterpret this and think they can incrementally build a skyscraper out of a shed.

That's what happened though? First humans built sheds, then we built 2-story buildings, then taller and taller, until we built skyscrapers. Obviously it wasn't a single structure, but we did have to evolve our thinking on how to build things, we didn't just start building a skyscraper before we built a shed.

You can't do that. A small bike shed is often just put some concrete blocks on the ground, and then build on top of them with wood. A correct house needs a stronger foundation at higher costs (sheds larger than bike shed are build the same way), but is still made of wood. A skyscraper is built with a very different foundation, and needs a steel frame that would not be affordable in a house. In between the two there are also building made of brick which allows building taller than wood. (and there are lots of other options with different costs - engineered wood is different)

Point is though eventually some system runs out of ability. It works different in programming from physical construction, but the concept is the same, eventually you can't make a bad early design work anymore.

Re: Systems Thinking

#85

Post by Terretta »

The paradox in post is resolved by limiting the planning to Russian doll like nested timeframes and scopes, upgrading from endless 2 week sprints and quarterly or annual "planning" to cycles within cycles, scoped to human magnitudes of time, and JIT re-planned at the Nyquist interval of each cycle by those at the corresponding level of the enterprise org chart who must also be domain leads with mastery at that level and who have retained ability to probe two levels below while practiced at bringing along at least one level up.

The 1970 Royce paper was about how waterfall didn't work, and most "Agile" is a subset of DSDM, each flavor missing a necessary thing or two whether working in large systems or growing them greenfield from nothing. But DSDM wasn't "little a" agile (and SAFE just isn't). There is a middle way.

If you like applying this stuff (e.g. you've chatted with Gene Kim, follow Will Larsen, whatever, sure, but you've deliberately iterated your approach based on culture and outcome observability), feel free to drop me a note to user at Google's thing.

Re: Systems Thinking

#86

Post by bluGill »

t43562 wrote: post #63

Big upfront designs are obviously based on big upfront knowledge which nobody has. When they turn out to be based on false assumptions of simplicity the fallout is that the whole thing can't go forward because of one of the details. Evolutionary systems at least always work to some degree even if you can look after the fact and decide that there's a lot of redundancy. Ideally you would then refactor the most troubles…

Big upfront design always tries to design too many things that should be implementation details. Meanwhile the things that are really important are often ignored - because you don't even realize they are important at the time.

Re: Systems Thinking

#87

Post by bluGill »

ako wrote: post #45
iafan wrote: post #28

Earlier quoted context omitted.

Everything that is touching hardware, for example. Bluetooth stack, HDMI, you name it. Everything W3C does. Go is evolving through specs first. Probably every other programming language these days. People already do that for humankind-scale projects where there have to be multiple implementations that can talk to each other. Iteration is inevitable for anything that gains traction, but it still can be iteration on sp…

I would expect them to build prototypes to iterate over the specs.

Prototype and the specs go hand in hand. Write a spec - prove you can implement it. Write an implementation - write a spec so we can talk about what is important (vs details of how you implemented it but someone else is allowed to implement differently) Often parts that are "obvious" are not implemented, or only the trivial version is implemented. You need to do both.

Re: Systems Thinking

#88

Post by hnthrowaway0315 » 

    > you lay out a huge specification that would fully work through all of the complexity in advance, then build it.
I have tried this a couple of time even for small projects ( a few sprints ), and they never worked out. I'd argue it never works out if you are doing non-system programming projects, and only has a theoretical non-zero possibility to work out for system programming projects, and perhaps a 5-10% to work out for very critical and no patch possible projects (like moon landing).

Because requirements always change. Humans always change. That's it. No need to elaborate.

Re: Systems Thinking

#89

Post by moring »

godelski wrote: post #47

> you lay out a huge specification that would fully work through all of the complexity in advance, then build it. This has never happened and never will. You simply are not omniscient. Even if you're smart enough to figure everything out the requirements will change underneath you. But I do still think there's a lot of value into coming up with a good plan before jumping in. A lot of software people like to jump in a…

The book "How Big Things Get Done" by Bent Flyvbjerg nicely answers all the concerns mentioned in this thread. I'll answer here to avoid littering replies everywhere.

> But I do still think there's a lot of value into coming up with a good plan before jumping in.

Definitely, with emphasis on a _good_ plan. Most "plans" are bad and don't deserve that name.

> be specified up-front, planned on JIRA

Making a plan up-front is a good approach. A specification should be part of that plan. One should be ready to adapt it when needed during execution, but one should also strive to make the spec good enough to avoid changing.

HOWEVER, the "up-front specification" you mentioned was likely written _before_ making a plan, which is a bad approach. It was probably written as part of something that was called "planning" and has nothing to do with actual planning. In that case, the spec is pure fiction.

> estimates provided

Unless this project is exceptional, the estimates are probably fiction too.

> and Gantt charts setup

Gantt charts are a model, not a plan. Modeling is good; it gives you insight into the project. But a model should not be confused with a plan. It is just one tiny fragment you need to build a plan, and Gantt charts are just one of many many many types of models needed to build a plan.

> before they even sign the contract for the next milestone

That's a good thing. Signing a contract is an irreversible decision. The only contract that should be signed before planning is done is the contract that employs the planners.

> Anyone who claims upfront specs are the solution

See bove. A rigid upfront spec is usually not a plan, but pure fiction.

> My approach, especially for a project with a lot of unknowns, is usually to jump in right away and try to build a prototype.

Whether this is called planning or "jumping in" is a difference in terminology, not in the approach. The relevant clue is that you are experimenting with the problem to understand it, but you are NOT making irreversible decisions. By the terminology used in that book, you are _planning_, not _executing_.

> after the 2000 pages specification document was written, and passed down from the architects to the devs

If the 2000 page spec has never been passed to the devs while writing it, it's not part of a plan, it's pure fiction. Trying to develop software against that spec is part of planning.

Re: Systems Thinking

#90

Post by nyrikki »

Warning, this post is not covering the ‘systems thinking’ that most of you will be expected to know in staff level jobs, it is using the term for up front design.

In a more typical modern sense systems thinking is more about relationships and wholes, rather than isolating parts, which is the traditional engineering approach.

While much of the base material on systems thinking will be based around cybernetics, it is really a complement to traditional engineering, used in parallel to identity more natural complexity boundaries and to help avoid confusion and accidental complexity.

Gregor Hohpe’s Architect Elevator is probably a good place to start on why this change in perspective is important and why investing in flexibility is crucial when there is uncertainty.

While you may have to accept this article’s definition in some groups, accepting the more modern definition will help you get jobs in places that are nicer to work.

This type of false dichotomy that is presented in the article is a warning that there is soft work to be done.

People mentioning mechanical engineering in this thread are possibly the people who may benefit most from examining the material. I encourage you to see if this is a path forward for your needs.

Post reply on HN