All of us test in production all the time (2019)

> The opinion of the article is that the catchphrase has become an excuse to build poor observability and in-prod testing tools, and we should be better about that.

Is that a problem that actually exists though? I've certainly never encountered it, as the types of teams the are strict about good pre-release testing are also the types of teams that are big on observability.

Most large companies encourage testing in prod. It's called split test. Clearly precursor to that is staging, and precursor to that is integration tests and whatnot. But none of that catches what you can with a/b test, and great monitoring.

Advice like this is good for applications that place a high value on stability and quality, which aren't deployed frequently, and which are monolithic (or rather, which are deployed monolithically--microservices that are deployed all at once are less a problem as it pertains to testing).

However, the further your application deviates from these invariants, the less good this advice becomes. And there are good reasons to deviate--many applications exist in a business context in which production bugs are often not a very big deal (e.g., social networks); however, the ability to iterate rapidly (especially the ability to test new features, etc rapidly) is paramount. In these contexts, it's entirely appropriate to trade off the ability to test in advance in exchange for the ability to iterate more rapidly--this includes things like moving to a microservice deployment cadence (different components deployed at different times to elide expensive cross-team deployment coordination moments). Microservices are harder to test to the same confidence interval because it's hard to fix the versions of all of the different components in order to assure reproducible tests, and the value of doing so is diminished because in practice any given combination of versions is likely short-lived since one or more components will be redeployed in a matter of hours or even minutes. Of course, unit testing and integration testing are still valuable in these contexts, but they aren't as valuable as they are in other contexts, so testing in prod (coupled with the ability to quickly fix bugs) is necessary to make up the difference.

So like everything in software development, "it depends".

Headline is unfortunate, and then the first part of the essay is spent justifying/walking back the title. The good part starts a few paragraphs down, at "We conduct experiments in risk management every single day".

He mentioned that he had a job doing that, but the company moved away from the framework without technical justification.

He could find another, but there would be no guarantee that the same thing wouldn't happen again.