Scrum is the new waterfall • Hacker News Board

Re: Scrum is the new waterfall

#101

Post by dragonwriter » Wed, Nov 11, 2015, 1:10 AM UTC

kuschku wrote: post #36

jdbernard wrote: post #33

Earlier quoted context omitted.

That's not true. Quality and agile process are almost orthogonal. It is entirely possible to develop extremely high quality software within an agile environment and very possible to develop extremely poor software using waterfall. Agile doesn't mean "every engineer working on whatever they wanted." Ultimately quality software results when developers place a priority on writing correct software and when management is…

> Working software over comprehensive documentation > Responding to change over following a plan These statements from the agile manifesto are directly opposite to any kind of quality software. Documentation and Code Quality are some of the most important features. Your software is not working if it doesn’t exactly match its documentation. You can live with that in web software, but you’d end in jail if you tried to…

> Documentation and Code Quality are some of the most important features.

Code quality is an important attribute, sure, but those don't say anything about it.

Comprehensive documentation of the bureaucratic sort of process documentation that I think motivated that line is not a deliverable, its an artifact of process; when it becomes understood as an end to its own rather than as something that should be used to the extent it is useful to achieve value in the end product, it becomes extremely problematic (I've experienced the kind of bureaucratic software development that this reacts to, and I can assure you that it does not contribute positively to quality, it contributes to management -- and staff -- spending lots of time worrying about and measuring and optimizing metrics that are unrelated to quality.)

> Your software is not working if it doesn’t exactly match its documentation.

Your software is not working if it doesn't deliver the expected behavior. Documentation of requirements is important to provide a consistent method of communicating between people (and across time and space) what those expectations are. And to the extent that it is necessary for that purpose documentation is a good thing.

But its important to keep the purpose in mind. Otherwise, you end up with excessive effort going into polishing documentation rather than building software.

> you’d end in jail if you tried to write security critical systems like that.

Sure, bureaucratically comprehensive documentation is extremely useful (and sometimes essential) for demonstrating compliance with process mandates. Such mandates, and the incentives they create, are also very often not helpful to the actual quality of software, including the aspects of software quality which motivate the mandates.

Re: Scrum is the new waterfall

#102

Post by matthewmacleod » Wed, Nov 11, 2015, 1:12 AM UTC

kuschku wrote: post #98

jdbernard wrote: post #78

Earlier quoted context omitted.

I'm very curious, what is your definition of waterfall?

The classical definition: 1. You research the actual requirements, including by testing some possible alternatives via mockups, etc. 2. You design the concept of your system, design APIs and interfaces. 3. You implement it. 4. You verify it matches the design and fulfills the requirements 5. You release it and maintain it. Most importantly, after Step 2 your APIs don’t change anymore. This is especially important as…

I appreciate that's the definition, but the problem that agile tries to deal with is that this almost never happens as you describe in practice.

In reality:

  1. You research the actual requirements
  2. You design the concept of your system, design APIs and interfaces.
  3. You implement it.
  4. You discover that the requirements are unimplementable because of some unforeseen design issue
  5. You make changes to your implementation, documentation and requirements to deal with this
  6. You discover that documentation regarding a third-party system is inaccurate and requires design changes
  7. You make changes to your implementation, documentation and requirements to deal with this
  8. You verify it matches the design
  9. You discover previously unknown acceptance requirements
  10. You make changes to your implementation… and so on.
  ??. You release it and maintain it

Even with skilled, competent developers and architects, no system ever turns out exactly as expected. The goal of agile development is to accept that and try to incorporate the concept of continuous delivery and adaptation into the core of the process – since it's required anyway!

Re: Scrum is the new waterfall

#103

Post by kuschku » Wed, Nov 11, 2015, 1:12 AM UTC

simons wrote: post #97

kuschku wrote: post #83

Earlier quoted context omitted.

Well, that’s why you start testing what you can do, what the issues will be, etc while still working on the plan. You don’t decide to build a bridge out of a specific steel without testing in models which kind of steel is ideal for your case, or if you might have to change the shape of the bridge. But in the moment that others will build projects depending on your implementation being stable – in the bridge example,…

But it's not just simply the "correctness" of a solution that changes, that 3rd party API you're having to interface with might not return data in the same way you were expecting, the other company who were going to deliver the grid component can't do it in time anymore, half the dev team have fallen ill and are out for 3 weeks. No amount of planning up front can ever be sufficient, agile gives us devices to manage a…

Well, sure.

But that’s the issue: What if you are writing the third party API, what if you are writing the firmware of a Washing Machine which is sold with lifetime guarantee*? (up to 99 years or death of buyer, whichever happens first), what if you write feature where every change has to be reviewed and costs millions?

People expect from quality software what they expect from a quality washing machine, meaning that they buy it once, never have to touch it again, and it will always work and never change. Except for some minor features added at some point.

But removing or changing features? Never.

This is true for users (remember how many people complained about Windows 8 unusual interface?) and for other developers (How many people go all crazy every time Google kills yet another service, or removes another API, or changes their API syntax?).

Sometimes it works, sometimes it doesn’t. But, effectively, I’m asking for perfect backwards compatibility. And that is rarely possible in systems where the software defines the documentation, instead of the other way round.

Re: Scrum is the new waterfall

#104

Post by parasubvert » Wed, Nov 11, 2015, 1:14 AM UTC

kuschku wrote: post #86

parasubvert wrote: post #84

Earlier quoted context omitted.

"you want a software that is backwards compatible so third parties interacting with you will never have to rewrite your software, you want a software that runs with minimal maintenance, but is easy to extends, etc. And Agile is ill suited for that." I really think you need to build a lot more software before you can make a claim like that. You're being fed lines from your professors. The vast majority of software, pe…

Well, if companies build their software depending on yours, before you have even written a single line of code, and if your software is supposed to run for decades, agile – which is better suited for constant change – is not ideal. And this piece of software exists. Governmental projects interacting with lots of insurance companies, for example. Like the Elektronische Gesundheitskarte – a concept of giving every Germ…

"Before the software was even finished, PoS software was able to interact with the API and insurances had sent out millions of cards."

Yes. There are hard constraints in many projects. If you know exactly up front what you need, you do not need Agile approaches to handle the uncertainty.

However, there is plenty of uncertainty to be managed behind this API.

"Agile is far more suited for constantly changing environments, where you also can just break support for any third parties depending on your API. But obviously, that’s not really quality software when every software depending on yours has to be changed every few years."

One doesn't have to change an API to evolve the software. Old school HTML and HTTP from 1992 still works on an evolving web.

"But obviously, that’s not really quality software when every software depending on yours has to be changed every few years."

APIs can be made to be backward compatible or versioned as they evolve.

Again, you presume the people asking for the software know exactly what they want. IF they do, then you are fine. It is very rare in practice except for simple or well-understood predictable domains. If the stakeholders do not know or agree on what they want (most of the time), something like Agile is appropriate to manage change.

Most APIs evolve a lot as parties learn more about the problem space, but this does not necessarily mean older versions automatically break. This really has little to do with Agile vs. Waterfall, it has more to do with the design decisions made for evolvability.

Re: Scrum is the new waterfall

#105

Post by kuschku » Wed, Nov 11, 2015, 1:14 AM UTC

matthewmacleod wrote: post #102

kuschku wrote: post #98

Earlier quoted context omitted.

The classical definition: 1. You research the actual requirements, including by testing some possible alternatives via mockups, etc. 2. You design the concept of your system, design APIs and interfaces. 3. You implement it. 4. You verify it matches the design and fulfills the requirements 5. You release it and maintain it. Most importantly, after Step 2 your APIs don’t change anymore. This is especially important as…

I appreciate that's the definition, but the problem that agile tries to deal with is that this almost never happens as you describe in practice. In reality: 1. You research the actual requirements 2. You design the concept of your system, design APIs and interfaces. 3. You implement it. 4. You discover that the requirements are unimplementable because of some unforeseen design issue 5. You make changes to your implem…

And now imagine a big company develops a software interacting with yours at point 2, and finishes it at point 5.

You’ll have to add backwards compatible wrappers that emulate an older API in 7 and later.

Or you can actually test with proof-of-concepts, with mockups, with other tests, before you even write the design.

Re: Scrum is the new waterfall

#106

Post by kuschku » Wed, Nov 11, 2015, 1:16 AM UTC

rjspotter wrote: post #95

kuschku wrote: post #72

Earlier quoted context omitted.

And I can tell you that even some cloud software, like the Elektronische Gesundheitskarte was developed with Waterfall. Often you have to design software interacting with dozens of other entities, all who are going to write software interacting with your system. And your system is supposed to be able to run for years or decades with minimal maintenance. Agile is far more suited for web apps, or small mobile apps, whi…

Healthcare.gov was also a healthcare application developed waterfall style and was an enormous failure on launch day because of it's interactions with 3rd party systems.

And now imagine Healthcare.gov was Agile and its API would change every month completely.

And then imagine how every insurance would complain and how, after 2 or 3 months, everyone depending on healthcare.gov would have stopped supporting it because after the third rewrite of the parts of their software interacting with healthcare.gov’s API they’ve gone mad.

Every time you interact with third parties you have to provide stable interfaces.

Re: Scrum is the new waterfall

#107

Post by matthewmacleod » Wed, Nov 11, 2015, 1:16 AM UTC

kuschku wrote: post #99

matthewmacleod wrote: post #94

Earlier quoted context omitted.

What you are suggesting – to plan, document, implement and test every feature one after another – is just waterfall for every problem separately. Well, isn't that the core of any software development process? We're ultimately delivering features, and they all require planning, documentation and testing. The development methodology used doesn't change that. 'Agile' is the key word – change will happen, and being agile…

I responded to your sibling comment with the definition of Waterfall. And every single comment defending Agile here was just "well, you can still document stuff before implementing it, and then verify it, and then maintain it". Which, effectively, is: Even if you’re agile, you can still do waterfall. Great...

Obviously I'm not making a particularly convincing argument…

At the end of the day, agile development is primarily about ensuring that you can respond to change quickly and with minimal disruption. Change occurs in all software projects, and agile tries to avoid some of the pitfalls that one can fall into by architecting an entire system in advance.

"Even if you’re agile, you can still do waterfall." is not quite what i'm trying to get across. Rather, it's that agile doesn't mean you can't provide documentation in advance if that is a specific requirement. Bear in mind that external APIs that must be stable are quite distinct from the rest of the system; indeed, it's a horribly design failure if you can't completely swap out your implementation without changing those APIs!

Re: Scrum is the new waterfall

#108

Post by kuschku » Wed, Nov 11, 2015, 1:18 AM UTC

gnoway wrote: post #93

kuschku wrote: post #85

Earlier quoted context omitted.

The issue is when other software is built upon those initial documentation of your system. Third party services depending on you having a stable API, for example. You can’t just break support for them, so what do you do in agile?

Don't break support for them? There are still requirements, there is still design. If you have a customer depending on your API, that is a requirement. Agile doesn't mean requirements go away.

And if your implementation never changes, but you just do minor changes – isn’t that the maintenance phase of Waterfall again?

Re: Scrum is the new waterfall

#109

Post by matthewmacleod » Wed, Nov 11, 2015, 1:18 AM UTC

kuschku wrote: post #105

matthewmacleod wrote: post #102

Earlier quoted context omitted.

I appreciate that's the definition, but the problem that agile tries to deal with is that this almost never happens as you describe in practice. In reality: 1. You research the actual requirements 2. You design the concept of your system, design APIs and interfaces. 3. You implement it. 4. You discover that the requirements are unimplementable because of some unforeseen design issue 5. You make changes to your implem…

And now imagine a big company develops a software interacting with yours at point 2, and finishes it at point 5. You’ll have to add backwards compatible wrappers that emulate an older API in 7 and later. Or you can actually test with proof-of-concepts, with mockups, with other tests, before you even write the design.

I think you missed my point there – I'm not describing an ideal agile process. It doesn't matter how many proof-of-concepts, mockups, tests and design documents you produce in advance. Change still happens.

Re: Scrum is the new waterfall

#110

Post by dragonwriter » Wed, Nov 11, 2015, 1:20 AM UTC

kuschku wrote: post #106

rjspotter wrote: post #95

Earlier quoted context omitted.

Healthcare.gov was also a healthcare application developed waterfall style and was an enormous failure on launch day because of it's interactions with 3rd party systems.

And now imagine Healthcare.gov was Agile and its API would change every month completely. And then imagine how every insurance would complain and how, after 2 or 3 months, everyone depending on healthcare.gov would have stopped supporting it because after the third rewrite of the parts of their software interacting with healthcare.gov’s API they’ve gone mad. Every time you interact with third parties you have to prov…

> And now imagine Healthcare.gov was Agile and its API would change every month completely.

Most methods that get the "agile" label attached to them focus on managing change by organizing work in such a way that, outside of externally driven changes that the organization can't control (which have the same effect regardless of development method) the parts that are done don't change much after they are released, to avoid widespread breaking changes.

So I don't see the point here, or how it connects to Agile.