Yagni (2015)

Yagni (2015)
martinfowler.com

I run a site that stores audio files. For the first half of its life, it simply stored the URL of the file on S3. I could have used objects representing audio assets with metadata to store the reference, but I didn't think I'd need it. When I was building another feature that _did_ need that functionality, the migration took far longer to write, test, and run than the process of building the asset objects feature and the feature I needed them for combined.

Additionally, the new system made it easier to debug customer issues. I didn't know that I actually needed this feature the whole time.

I see YAGNI fail when it's used an excuse to not think about the future at all, rather than a reason to delay building parts of a system that aren't needed yet. Relative to building software, thinking about it is cheap.

I usually think about:

    * how needs might change in the future
    * what would have to change in the current system 
      in order to satisfy those future needs
    * how difficult changing the system to meet 
      those future needs would be.

If so, I hate to say it, but doesn't YAGNI still apply here?

This was, essentially, a scaling problem: the O(N) time-cost of querying metadata for N S3 objects was too high. You still could have written code to query that metadata out the "naive" way anyway, and it would have worked for low N. But it wouldn't have worked "at scale."

But there are two [point five] ways to solve a scaling problem:

1. When you need to scale, write more code, customizing your logic to make it more performant, add locality or caching, etc. (This is what you did, and the way most software engineers think.)

1.5. Anticipate the need to scale, and write code "the more performant way" in advance. (This is where the YAGNI admonishment comes from.)

2. When you need to scale, attempt to find an infrastructure-level solution that involves writing no code. This is the way ops people tend to think, since they don't write code (or at least, they trust their knowledge of infrastructure solutions better than their coding abilities.)

An example of approach #2, in this case, would be something like "put Varnish between you and S3, and configure it to only cache HEAD requests (and to synthesize HEAD response cache entries from proxied GET requests, without caching the GET response itself.)" Then your existing O(N) S3-metadata-querying code would—after warming the cache—suddenly be faster; it would probably be fast enough to answer whatever sort of debugging questions you'd like.

The reason people say "YAGNI" is that, often, an ops person can take your developed software as a black box, and solve its scaling problems without touching the box. And this is often the optimal way to solve these problems: you probably can't write a caching layer for your web app, inside your web-app's process, that will work half as well as Memcached. Or a logging system that will work half as well as rsyslog. Or a web server that will work half as well as Nginx. Neither can the people who develop packages for your programming language's ecosystem. The real experts in "what you need for production-scale" converge on an infrastructure component and develop that, rather than contributing to FooLang's logging implementation.

And, since these scaling problems can be solved without ever touching your code, it's especially silly to try to anticipate scaling problems you might have and solve them in your code, early. The only scaling problems you should worry about at design time are the ones that can't be solved by IPCing infrastructure components together. (Such as, for example, the cost of a huge number of concurrent threads. Erlang/Akka/etc. exist because they solve a particular scaling problem that is intractable to anything but a process-architectural solution.)