This is a good question. The answer is so that we can amortise the expense of feature development. Side note: This includes bug fixes. A bug is just a feature that you expected already worked. A bug is different from a "software error", in that such errors will impact you when you next work on the code. It is often possible to fix bugs without fixing the underlying software error and vice versa.
Having a "flexible" code base (to me) means being able to change the code easily. That change could be to add new functionality, or to refactor the design, or both. "Flexible" (to me) does not necessarily include having actual facilities to do anything.
As an example, your normal "Hello, world!" C program is very, flexible. It's easy to modify. A similar "Hello, world!" written using a large framework is less flexible because you have more constraints on what you can do, and how you can change the design, even though it has more facilities.
Adding flexibility does not mean adding more facilities to the code. Often it means removing unused facilities from the code in order to make it simpler. IMHO, this is the true distinction with YAGNI. We remove YAGNI code in order to increase the flexibility of the code. Sometimes people are tempted to write wonderful elaborate designs for their code in order to ensure that it's easy to do something in the future. This is a classic case of YAGNI. We want to remove that code in order to increase flexibility (because often new requirements move in a direction that is opposed to the original design).
However, I sometimes wish that there was a cute acronym for "I Actually Need It Now" (suggestions welcomed :-)). In this case, let's say we've written some code and it's awkward, but it does the trick. A little while later, we end up doing the same thing. It's relatively easy to copy and paste our previous awkward code. Then we have to do it again. In those cases, you've found that you do need it, and you're adding complexity to your code base every time you copy and paste the same awkward code. It's beneficial to build a nice system to make it easy to do it. Next time, not only will it be quick to add, but it won't add unnecessary complexity to the code base.
You've probably heard of the "rule of 3". Of course, it's a rule of thumb, so you have to use your own judgement, but the idea is that if you have to do it once, then just do it. You can't generalise what problem you are actually solving because you just did it once. There's not much sense in agonising over the "ultimate" design, because you are likely to get it wrong anyway. But by the time you've done it 3 times, you've got a pretty good idea of where this is going. At that time you should invest in an appropriate solution. The solution will likely increase the architectural complexity, but it will simplify the interfaces and will slow down the increase in overall complexity.
Again, 3 times is a rule of thumb. Sometimes you know right away that you need something more complex. Sometimes it takes you a lot more than 3 times before you can really wrap your head around what direction is best. So a better rule is: delay making design decisions until you have enough data to answer your questions.
And to roll it all back to the beginning, YAGNI happens when you make decisions before you have the data to support your decision making. This can be both with user features and with design. By delaying these decisions until we have good data, we keep our options open and the system remains "flexible". But if we do not make the decisions when we do have enough data, the system becomes complex because it lacks cohesion. We have to act promptly to address those issues, or else we will have a very uneven development experience. "Hard" things may take a short amount of time, because we didn't build all the infrastructure we needed. "Easy" things may take a long amount of time because we suddenly need to build the infrastructure that we didn't build before.
And that's what I mean by "amortising" the expense of feature development. We don't spend a lot of time up front to build a complex, but inflexible system. However, we spend time intermittently, to maintain flexibility and to keep the overall cost down. Even more importantly, businesses depend on the predictability of development. By maintaining a high degree of flexibility, we allow the cost of solving problems to approximate the complexity of the problem. Without having a flexible code base, the cost of solving the problem is often dominated by the complexity of the code base instead of the complexity of the problem. This leads to management not being able to "trust" development and often leads to project cancellation.
Another aside: I often say that upper management is the most dangerous part of the team for a project because they are the ones that cancel the project. If they do not have a good feel for a project, they may very well cancel it for extremely poor reasons. Thus, aligning the project to the expectations of upper management is one of the most important parts of software development.
I personally don't like doing development work that isn't tied to direct economic value. This includes refactoring. Refactoring may contribute to indirect economic value (by reducing development time int he future), but it is risky work. As I mentioned in the previous paragraph, upper management depends on the predictability of development. If you randomly (from their perspective) say, "We're going to do some feature sized work, but it's not going to result in any different functionality", it separates them from the logic of the development. They may tolerate it, but it makes planning difficult.
Instead, I prefer to spread that kind of work out and to "same size" the work. This is harder to do from a design perspective because you have to find small goals to achieve and to work piecemeal. You need to have good communication with your team and good buy-in for the process, so that each team member takes the appropriate opportunities every time they touch the code. In short, it makes the development process more difficult, but the advantages are many. In that way, I do exactly what you suggest: I only work on code that is adding customer value. But I intentionally take a little bit of extra time in each story to find ways to improve the flexibility of the code. I also spend time each story to add facilities (which I've called "capabilities" previously) where appropriate. Occasionally you get the, "OMG, we need to do a big refactor", but if you have been diligent in maintaining flexibility, then it should limit the amount of time that is necessary to move in the correct direction.