Ask HN: Do you test in production?
51–60 of 75 posts
Re: Ask HN: Do you test in production?
#52Everyone tests in production. Some people also test before production! Some people try to NOT test in production, but everyone does test in prod in a very real sense because dependencies and environments are different in prod. I think the question was "Do you INTENTIONALLY test in production"
Yeah. What some people mean by “we don’t test in prod” is that we have very high confidence in all code before it goes to prod. And good for them! But what most people mean is “code goes from zero to affecting all users and developers have little or no access to data about how that’s going.” Bad. No matter how many unit tests you have.
Re: Ask HN: Do you test in production?
#53If you are lucky you have internal users who use the product in production. Then if you are lucky you have a group of external power users who appreciate getting features first and understand that there is a risk of bugs.
Most software probably falls in this category of “we could test more but at some point our users would rather get the product with bugs than wait”.
Whether it’s staged rollouts, feature flags, it’s the same thing. It’s mitigating risk when testing in prod. It’s the best bet.
Some software obviously falls into the category that can’t have serious bugs for any users. Then you just have to keep the software so simple that you can be confident it works.
Re: Ask HN: Do you test in production?
#54I don’t recall testing in production ever being taboo in the 90’s.
Re: Ask HN: Do you test in production?
#55Currently not, but there was a project where we had to "develop" in production. I was coding a IoT adapter for a building automation system. We sure had development machines but when we first tested our code in real env we noticed they were slightly different version. So there was no other way than to ssh into our machines and use Vim to make code work and then replicate the changes on own computer. Fun times but don't really miss the stress of messing up something on real building.
Re: Ask HN: Do you test in production?
#56Anytime you need to talk to a third party API, you need to test in prod. Some people have sandbox apis. They are generally broken and not worth it. See eBay for super in depth sandbox API that never works. You can read the docs 100 times over. At the end of the day, the API is going to work like it works. So you kind of “have to” test in prod for these guys.
cough CME cough
Re: Ask HN: Do you test in production?
#57Lots of ways to test in production. IMO the way you are suggesting – injecting synthetic data into prod – is the worst of both worlds. You aren't actually testing real world use cases, and end up polluting your prod environment. Some common ways to go about it: - Feature flags: every new change goes into your codebase behind a flag. You can flip the flag for a limited set of users and do a broader rollout when ready.…
IMO your first two suggestions aren't actually a form of testing.
If you've rolled out a feature to real paying customers (not beta testers), and find that it's broken, you don't get a pass because you were only "testing".
Re: Ask HN: Do you test in production?
#58At my workplace which deals with millions of customers.
What we have is essentially a segmented clone of prod. It's a 1:1 copy of prod with real data flowing through.
We use feature flags to enable-disable features. This way when our devs ship code to prod - it first lands in this segmented clone.
Then incrementally changes are propagated from this segmented area into actual prod-prod.
Re: Ask HN: Do you test in production?
#59At my workplace which deals with millions of customers. What we have is essentially a segmented clone of prod. It's a 1:1 copy of prod with real data flowing through. We use feature flags to enable-disable features. This way when our devs ship code to prod - it first lands in this segmented clone. Then incrementally changes are propagated from this segmented area into actual prod-prod.
Is this different from a staging environment?
Re: Ask HN: Do you test in production?
#60Just to play Devil's Advocate and be argumentative, what is the point of testing in production when your development/staging environment is guaranteed to be identical to your production environment?