Earlier quoted context omitted.
Ok, there's actually a phrase from the article that says that, so we'll put that up there and see what happens.
For us curious latecomers, what was the original title?
"I test in prod (2019)"
All of us test in production all the time (2019)
121–130 of 164 posts
Re: All of us test in production all the time (2019)
#122Earlier quoted context omitted.
Wholeheartedly agree to everything you’ve said. I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing. For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a…
> They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes. If people are finding tests unnecessary extra work, maybe they're right for their project. Most jobs are just "ship feature asap, try not to break anything, repeat". Most managers don't care about code quality as much as speed of delivery. Most…
Sorry, I have to completely disagree with this. This is a horrible, but often used, excuse.
In my experience these coders are doing exactly what you would do in a unit test but instead they are doing it manually by adding printlns and testing various values on the command line, in a repl, a notebook, or a main. Instead of manually adding these things every time you want to test your work, if you just added some basic unit tests it would speed up your delivery. Not only that but you can't "forget to test that" because you're forced to write down the test cases instead of deleting them when you're done.
If you add up all the time you spent adding printlns to your code, testing various input values in a main method, researching inside your codebase to see what could fail, debugging your ETL because you forgot about an uncaught error or a different kind of null in this column that only came up after it processed 3TB of data (real example), etc. I guarantee you would save time if you just implement some basic automated tests instead of testing in prod or staging.
Re: All of us test in production all the time (2019)
#123I love articles like this because it's so easy to just add that company to a list of places to never ever work. I did read the whole article, btw. It's an absolute clickbait title that the author doesn't really mean, and after the article spends a lot of time diffusing the clickbait title it really boils down to, "This is hard, so I give up." It's true that many--if not most--companies operate this way without ever a…
This works great if you're building something with a tightly controlled API. If, however, your configuration space grows to an even middling size, it no longer becomes feasible to do much of this validation across the configuration space. A good example is any system where the user can customize system aspects. Do you run all of your integration tests across the full configuration space? Additionally managing configu…
I didn't say that any of this was simple or non-trivial. Again, it depends on your priorities and your values as well as your company culture. In fact, I specifically said that testing systems are hard and provided examples of how hard systems are to test. Do you think that Cassandra is a tightly controlled API with a small configuration space?
You seem to feel like close enough is good enough. And that's the cause of the problem I'm trying to address here. Does it really matter if you don't get a notification when someone messages you on Facebook? Or if you get two notifications? Is that particular problem worth testing every possible Kafka configuration? I think that you are saying is no, it doesn't matter.
But I'm arguing a different point. I'm not arguing about whether the testability of any individual feature is important. For obvious reasons: some features really just aren't that important. But not being able to do that, and actively choosing not to understand that system is a symptom of a far deeper problem. When a company makes the choice you have just described, the company has decided to accept that they can't, won't, and will never fully understand their own systems. It's often not a conscious decision, it's a decision made by habit, policy, and culture, which is what's so subversive about it. People don't make big-picture decisions to intentionally have a system that is unknowable/untestable. People make small decisions just like the ones you are talking about that make systems that way. And it's the practice of letting lots of disconnected people make the small decisions of what does and doesn't matter, what is and isn't worth it that destroys systems.
Systems are hard, and I agree with that, but systems are made even more so by bad process.
The being old analogy didn't seem to resonate with you, which is fine. But let me ask you a question about a system.
You have a database. It gets backed up every night. Or maybe every hour. Your job is to take snapshots and store them because that's what you're supposed to do. Yeah, I know, that should be or can be automated. Whatever.
The big picture system and purpose is that you are supposed to be able to recover from a hardware failure/data loss. But that's not your problem. Your problem is that you have to back up the database manually every day. The data team only tests restoring backups from dev to dev instead of prod to dev. Because reasons. Because it's hard.
That type of backup system checks all the boxes you're supposed to check when you get audited. Or at least enough to get through it. But when you really need to understand the system, it fails for all kinds of reasons and people are sitting around looking at each other saying, "well I did what I was supposed to do."
Individuals sitting around making isolated, disconnected decisions like the ones you're talking about (i.e., it just isn't worth it; it's not feasible; it's hard) compound in organizations and create the kinds of systems you don't want to deal with. You're making your own hell here. You seemed to have missed that key point in my earlier comment.
Laziness is a good trait in an individual programmer. But laziness is the absolute death of an organization. Agile is really just distributed, organizational laziness. That's what creates horrible, unknowable systems.
Conflating test/experiment with what the original article claimed to be talking about (and then later walked back) is borderline disingenuous. No one is talking about A/B testing or intentional experiments.
The article is talking about rolling the dice in production deployments and claiming that's fine and something to be proud of. It isn't fine, and it's not something to be proud of. She's the CEO. She should fix her company instead of being proud of how bad it is.
A lot of what we're talking about here is a matter of perspective. And that is the problem I'm taking to task both with you and with the article.
Re: All of us test in production all the time (2019)
#124Earlier quoted context omitted.
Glenford Myers in "The Art of Software Testing" said: "Testing is the process of executing a program with the intent of finding errors." The preface to this is even more enlightening: "When you test a program, you want to add some value to it. Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors. Therefore, don’t…
I like this quote from Kent Beck: “ It is impossible to test absolutely everything, without the tests being as complicated and error-prone as the code. It is suicide to test nothing (in this sense of isolated, automatic tests). So, of all the things you can imagine testing, what should you test? You should test things that might break. If code is so simple that it can't possibly break, and you measure that the code i…
When I’ve gone full TDD on projects, I’m always surprised to find there’s a power law distribution or something on failing tests. Most of the tests I write never catch a bug in the life of the software. Something like 90% of the value of a test suite could be achieved with only about 10% of the tests. Of course, the trick is figuring out which tests are going to repeatedly fail ahead of time. But there’s something interesting down this rabbit hole.
I wonder if it would be worth collecting per-test stats through the life of a project to explore this.
Re: All of us test in production all the time (2019)
#125Earlier quoted context omitted.
I like this quote from Kent Beck: “ It is impossible to test absolutely everything, without the tests being as complicated and error-prone as the code. It is suicide to test nothing (in this sense of isolated, automatic tests). So, of all the things you can imagine testing, what should you test? You should test things that might break. If code is so simple that it can't possibly break, and you measure that the code i…
When I’ve gone full TDD on projects, I’m always surprised to find there’s a power law distribution or something on failing tests. Most of the tests I write never catch a bug in the life of the software. Something like 90% of the value of a test suite could be achieved with only about 10% of the tests. Of course, the trick is figuring out which tests are going to repeatedly fail ahead of time. But there’s something in…
You'd get a lot of the value by just being willing to delete tests once it's clear they're not serving any purpose. Sadly a lot of people would rather see a high test coverage percentage than have an effective test suite.
Re: All of us test in production all the time (2019)
#126Earlier quoted context omitted.
This works great if you're building something with a tightly controlled API. If, however, your configuration space grows to an even middling size, it no longer becomes feasible to do much of this validation across the configuration space. A good example is any system where the user can customize system aspects. Do you run all of your integration tests across the full configuration space? Additionally managing configu…
I didn't say that any of this was simple or non-trivial. Again, it depends on your priorities and your values as well as your company culture. In fact, I specifically said that testing systems are hard and provided examples of how hard systems are to test. Do you think that Cassandra is a tightly controlled API with a small configuration space? You seem to feel like close enough is good enough. And that's the cause o…
> I think that you are saying is no, it doesn't matter.
No I'm not saying that. I'm saying that the best way to prevent that isn't always to have a staging environment that mirrors production as well as you can.
> Individuals sitting around making isolated, disconnected decisions like the ones you're talking about (i.e., it just isn't worth it; it's not feasible; it's hard) compound in organizations and create the kinds of systems you don't want to deal with. You're making your own hell here. You seemed to have missed that key point in my earlier comment.
No, this was an intentional decision by the organization, that the organization shouldn't continue to invest time in solving the problem this way, because after significant effort expended by the organization, the conclusion of the people who the organization asked to investigate the problem was that solutions would not be feasible and would not improve things. You're acting like these decisions are always made in a vacuum. They're not. Often smart organizations investigate and make decisions at the level of leadership.
> Conflating test/experiment with what the original article claimed to be talking about (and then later walked back) is borderline disingenuous. No one is talking about A/B testing or intentional experiments.
Are you sure?
FTA:
> We conduct experiments in risk management every single day, often unconsciously. Every time you decide to merge to master or deploy to prod, you’re taking a risk.
> A healthy culture of experimentation and testing in production pulls together all three.
Canarying is just testing in production, but you have processes and "guardrails" (quoting the article) to make sure that it is done safely by default.
For the record, I work primary on reliability and release/experiment, and so I'm well aware that being explicit about your decisions is vital, as is knowing the tradeoffs involved. That's why pretending that you don't test in prod is a bad idea, because you almost assuredly do. That's what the article is saying.
Edit: As for Cassandra, it looks like they have system bugs caught in production, so I'm not sure what your point is (https://issues.apache.org/jira/projects/CASSANDRA/issues/CAS...)
Re: All of us test in production all the time (2019)
#127Earlier quoted context omitted.
> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similar…
As QA I think testing, test environments, test ressources should be first class concepts in software. Building whatever "modern" architecture twice, and simulating traffic is not easy but possible, at least to an extent. You need to get that extra license for any piece of software for testing during procurement, and provide means to create test ressources, like typically test users as needed. Sounds trivial, but is o…
It's not the architecture, it's the representing all the states in prod in testing.
Not saying it's impossible, just not always feasible to have it actively "perfect".
I've found that an "ok" testing envrionment and solid unit testing / monitoring tends to do better.
And then b/g deploys and canaries :/
Re: All of us test in production all the time (2019)
#128Apropos of this in the context of hiring, one of the biggest green flags I'll attach to someone's CV is any combination of infrastructure/operations, and application development, at the same job. Since the 1990s I've worked around ISP, hosting, and cloud firms. Many have a core of general purpose people that can't help themselves but have one foot in both graves (we call this DevOps or SRE now, but those are new labe…
Wow, you just described me. Except I'm about 4 years into my career. Glad to hear that I have a type. It is oddly reassuring in a manner that I can't currently articulate.
Most places I apply I just don't hear back from. Places I have worked had no idea how to use me effectively. I've applied for very few jobs because of this (on the spectrum, have trouble with rejection). I do a lot of free or underpaid work or just do something that fascinates me. The poverty sucks but I love that I get to pursue so many interesting things.
I have faith that I will find somewhere that is a good fit one day. In the meantime, I get to explore my interests and grow. Thanks for this post, it gave my hope a much needed boost.
Re: All of us test in production all the time (2019)
#129Apropos of this in the context of hiring, one of the biggest green flags I'll attach to someone's CV is any combination of infrastructure/operations, and application development, at the same job. Since the 1990s I've worked around ISP, hosting, and cloud firms. Many have a core of general purpose people that can't help themselves but have one foot in both graves (we call this DevOps or SRE now, but those are new labe…
Wow, you just described me. Except I'm about 4 years into my career. Glad to hear that I have a type. It is oddly reassuring in a manner that I can't currently articulate. Most places I apply I just don't hear back from. Places I have worked had no idea how to use me effectively. I've applied for very few jobs because of this (on the spectrum, have trouble with rejection). I do a lot of free or underpaid work or just…
Your self-description aligns very much with my own. Feel like we might get along well. Would love to chat. Shoot me a PM if you're willing.
Cheers, mate. There definitely is a place for people like us in the world.
Re: All of us test in production all the time (2019)
#130Earlier quoted context omitted.
Wholeheartedly agree to everything you’ve said. I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing. For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a…
> They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes. If people are finding tests unnecessary extra work, maybe they're right for their project. Most jobs are just "ship feature asap, try not to break anything, repeat". Most managers don't care about code quality as much as speed of delivery. Most…
Then, may I present the argument that, in general, you should be testing if for one reason: to get practice writing tests for when you actually have a project that you need to write tests in (edit: according to your metric).