Yagni (2015)

[1] Article from Dr. Dobb's gives a very high-level overview of the kind of data that is available.

[2] is one of the foundational books in this area. Its analysis of testing effectiveness is fascinating because it's so revealing. For example, comparing defect rates in projects that use TDD vs. non-TDD; the effectiveness of static analysis; etc.

[1] http://www.drdobbs.com/225701139 [2] https://www.amazon.com/dp/0132582201/

> YAGNI does nothing to stop you from misunderstanding the problem.

A lot of times it does because you usually gain more information about your users and problem domains as time goes on. So if you build a feature when you need it a lot of times you understand the feature better than before hand.

But I think we're all in agreement that if you understand a domain very well, think there is a very high chance a feature will be required, and it's a good deal more expensive to build later than now than you should build it now.

The features aren't random. They're usually related to other features. They might be variations on a feature. For example, what if the user doesn't like the apps fonts and colors? Let's let them customize the look and feel to whatever they want. Or we've made this app that let's you send text messages to people, but what if it needs to be sent to an international number? Suppose this feature is right now unnecessary because the app is just for sending text messages among employees at a company that is situated in Wichita.

These aren't crazy features, but they're unnecessary and may never be necessary. As to "why do their managers and technical leaders allow it," often they are the very people who spearhead it. Or in my case, our team was embedded in a non-software company, and our leaders didn't understand exactly what we did, so they were hands-off.

The best examples of YAGNI are usually when you say, "I'm adding this data structure even though we don't need it. We're going to implement feature X next month that does need it, so I'm adding it while I'm here". It happens quite a lot. And then feature X gets cancelled and you end up with this weird data structure in your code that doesn't do anything except complicate the code base.

There are other, more subtle issues as well. For example, you might want to add an iterator for something, even though your current task doesn't need iteration. However, you have strong feelings about how the iterator should work, so you add it because it helps you take control of the development. In those cases, it's easy for the rest of the group to say, "Thank you for that nice spike. However, that functionality is YAGNI right now. Let's take it out and we can discuss how it should operate when we're working on that feature". Basically, it's a way to stop people from bullying by intentionally creeping the scope of their work.

And finally, sometimes you honestly think something is needed and someone points out that it isn't necessarily true. It might be needed, but without getting more details it's hard to know right now. In those cases it's often better to remove the code as "YAGNI", keeping the design simpler. This allows it to move in multiple different directions -- some of which may obviate the need for that code. If you leave the code in, then it becomes a bit of a manifest destiny -- you write your code consistent with that extra complexity, even though you may not need it at all.

The above hopefully answers your actual question. Below is a very long discussion of my opinion of how XP development should ideally be practiced with respect to design. I hope you find it interesting -- or at least enough of a departure from what you've seen before to make it interesting for you.

Avoiding YAGNI, TDD and avoiding up front design all dovetail in an XP style development team, but my experience has been that the dovetail becomes a monstrosity if you are not actually doing XP. This is possibly why you seemingly have a negative opinion of it.

The main thing I tell people when I'm explaining XP is that "no big design up front" is not usually discussing the kind of design that they are thinking about (for example a week or so of trying to understand how to implement something and coming to agreements on design by drawing diagrams or writing example code). It's more about avoiding the kind of old-school design phases that lasted 6 months or more. When I was first starting out, we had minimum 18 month development cycles (and at one place we had a 5 year development cycle!). So we did 6 months of requirements analysis, 6 months of design, 3 months of development and 3 months of QA. But in reality, when you started development, you realised that the chunk you bit off was so big that it was basically just fiction. It turned into 12 months of patting ourselves on the back for being brilliant analysts and designers followed by 3 months of hacking and 3 months of absolute pure panic.

XP attempts to reduce the size of these phases and to distribute them more evenly into the development effort. It does not attempt to avoid them. With our old-school ways of doing things, our analysis and design was completely wasted because it was too disconnected with the actual development. With XP, you try to (as much as possible) marry the analysis, design and coding.

Of course that's usually not possible. One of my biggest problems with "Agile" teams is that they do so little analysis up front that they have no idea what they are building -- and it ends up being the same big hack we did in the old days. They avoid the panic because they just don't do QA and bury their heads in the sand thinking that their hack-a-thons are producing brilliant solutions.

Ideally you have a good 2-3 month backlog of stories that have been thought out enough that it only takes a day or so to implement each one. The key is that these stories are changeable and you "groom the backlog" at least once a week to see how your understanding of what you are building has changed since the last grooming session. By the time you commit to a sprint (very complicated discussion, but I like 2 week sprints), all of the stories should be implementable with only minor details remaining. While a "story" is a "reminder to have a discussion", if you haven't had the discussion before you started writing code, you are pretty much screwed.

Additionally, there is no way you can write stories of that size without having a basic understanding of the overall design that you are going for. Again, if you get to the point where you are going to put the story into the sprint and the developers don't know (or don't agree) how it should be implemented, then you need to insert a spike story instead.

This handles the "strategic" level of design. There is no "design phase". Rather it is spread out over development and updated over and over again. Any overall guidelines should be captured in a "coding standard" (which is not just a document describing how to indent your code, but rather a document that describes the overall "flavour" of design and the kinds of typical solutions that are preferred).

"Tactical" design is dealt with very differently, though. The idea is that code is design. It's just a very formal description of the design. Importantly, it's the only design that counts. It doesn't matter what your design diagrams look like if your code is not implemented that way. In most XP teams, diagrams, etc are usually considered transitory artefacts. The design is embedded in the code and you should write the code in such a way that the design is easy to see. That is not to say that you shouldn't draw diagrams, or utilise other kinds of design tools! It's just that you shouldn't attempt to persist them in any volume or (heaven forbid!) generate your code from them. The maintenance is just too expensive and error prone. Get used to understanding design by reading code -- you have to do that anyway.

The design aspect of TDD (if you are reading it as Test Driven Design) is a particular method. I could easily double or treble the size of this post and still leave you none the wiser. However, the basic thing to understand is that test code is very hard to write if you did not provide enough access in the design to write the test. Very, very often, as you expand the system the access that you provided for the tests makes it easy to access from the production side. Similarly the design is malleable because there is less coupling, though sometimes at the cost of cohesion. One very wise person (whose name I completely forget) suggested that a good way to understand how this works is to remove all global variables from your code. After doing this, start removing as many instance variables from your objects as you can. Once you've done that, consider ways of removing even more. Observe how that impacts your ability to easily write tests and to refactor the overall design of the system.

Although you always have to consider your strategic moves in design, coding (and especially TDD) is about tactical moves -- If I do X, then it allows me to do Y. At some point, you realise that this gives you an ability to move strategically. But that strategic decision is almost always better to discuss outside of the code (or in a spike), because it affects the whole team.

One of the main goals for XP style development is to end up with a design where it is easy to change direction. This is great for organisations where changing direction is important. Sometimes you are in a big stodgy company that literally doesn't care what the software does and never asks you to change directions. In that case, I probably would not worry about it so much. I've worked in groups like that and it doesn't matter what kind of lode stone you strap around your neck -- you won't get into problems. But in most places (and especially startups!) being able to change directions at the drop of a hat is extremely valuable -- sometimes to the point of rescuing a company. When you have discussions like "I know we asked you to do Y 3 weeks ago, but we were completely wrong. We need Z now. Is that possible?" and you reply, "Yep. No problem. Changing software is our job", then you know you've got it right. Getting it right is super hard, though ;-)

I think one of the reasons you don't see people talking about strategic ideas of design in XP is simply because they take it for granted. If you don't have a strategy for what you are doing, you are doomed. This is one of the reasons why I don't really like doing outside-in approaches, except if I'm spiking. Certainly, I consider advice to always do outside in as being exceptionally bad advice. It's a great technique to help you discover what you need to do, but I don't recommend it otherwise.

Anyway, I realised I launched a book off of your 2 sentence off-the-cuff comment, but I often find that people take popular misconceptions of good ideas and then treat the good idea as if it were the equivalent to the misconception. Now, I'm not going to say that my interpretation is mainstream at all. I'm also not Kent Beck, so I have no authority to say what XP is supposed to mean, but especially when you think an idea won't work, it's easy to choose the worst possible interpretation of it as being the most authentic. Hopefully this gives you food for thought.