The taboo is when you only test in production. At the very least you should manually try out your app after deploying a change. As far as automated integration tests in production, it is simple as identifying which tests are prod safe, and marking them. That really depends on the app, but in a web app it generally means all the GET requests, plus some of the others.
Ask HN: Do you test in production?
31–40 of 75 posts
Re: Ask HN: Do you test in production?
#32Hi! I wrote the referenced Segment post! Happy to answer any questions.
The way we did it safely is just as you say: creating fabricated users/organizations/configurations with data generators injecting into the system.
Faking data to look realistic is always challenging, but we used this cool library written by an early segment engineer: https://github.com/yields/phony
Not perfect but works well enough! And it's super simple. :-)
Re: Ask HN: Do you test in production?
#33Anyone with CI/CD is testing each deployment in production, right?
Re: Ask HN: Do you test in production?
#34What I've done in the past is to write a test that runs every five minutes in production, accessing the APIs like a user for the most common app flows. It provided a great way to be sure the app was genuinely working.
That did require having multi-tenancy support, and there was a need to suppress some security features by whitelisting the IP of the test app.
Re: Ask HN: Do you test in production?
#35In most of the systems I worked with the ACID database is the source of truth. So I carefully (there is framework support to reduce errors) run tests without committing the open transaction. Not recommended, but sometimes database copies don't surface the actual problem.
Re: Ask HN: Do you test in production?
#36I've been testing in prod for 20+ years, here are the best practices I suggest:
tl;dr: Safety comes in the form of confidence that you will know right away when something has gone wrong and can quickly recover from it back to the last known good state.
1) Observability is key. You can't test in prod unless you have really good metrics and monitoring in case you break something. It's also the only way you'll know the test worked. So that has to come first.
2) Automated deployment and rollback. You need your deployments to be fully automated as well as rollback. That way if something goes wrong you can quickly back out the change. It also means that devs can roll out smaller changes, because they don't have to amortize any deployment overhead. If a dev knows it will take 30 minutes minimum to deploy, they won't do it as often. Smaller deployments more often mean smaller blast radii.
3) Automated canaries. Once you have 1 and 2, you can fairly easily build 3. When code is checked in, have it automatically deploy and receive a small portion of traffic. Then have it automatically monitored and compare metrics. If the metrics are worse on the canary, roll it back.
You don't need to automate step 3, it's just a lot easier. But you can totally do step 3 by hand as long as you have 1 and 2.
These steps apply to stateless systems, but they can easily be applied to stateful systems with some small changes. With stateful systems you can still do canaries. But you have to add an abstraction layer between your business functions and their datastore (but you're doing that already right?). In that abstraction layer is where you add the coordination to keep data in sync during transitions from one data store to another (when doing schema changes for example). Or if you're changing the way you write to the data store in any way, so that you can write to both new and old and read from new and old without the code being different between them.
And then lastly you start adding in chaos engineering [0]. If your systems can automatically recover from errors in production, then it can automatically recover from bad deployments.
Re: Ask HN: Do you test in production?
#37I put my new feature behind a beta flag or experiment flag. If the flag is off for a user, they don't see it.
Then I turn it on just for the user I test with in prod. Then I test in prod.
When it's time to enable the feature for the rest of the users, the same system let's me slowly dial up which users can see the feature. This separates deployment from launch, which is also a great best practice.
Re: Ask HN: Do you test in production?
#38Generally, no. I have been known to point my local instance to production database where I am now as it's easier to get the dataset where an error occurs. I don't do anything that requires changing the data, strictly selects and views. I make a point to switch it off production ASAP.
I would prefer not having to do that at all though.
Re: Ask HN: Do you test in production?
#39In most of my projects I made sure that there was always available a recent (1) copy of the whole Production DB - this is mostly used be able to replicate erroneous behaviour in a controlled environment.
But it is also useful to get very close to "test in prod" without actually risking anything.
Actually executing data-changing code for testing is actively discouraged, though.
1) the current system takes a snapshot of the production db at the end of the day and uses it to repopulate from scratch this "staging" environment. In past cases I had to accept less frequent updates, though.
Re: Ask HN: Do you test in production?
#40My org has done a bunch of what's already covered here. We have a bunch of customers (SAAS), and though we have a good idea of what's going well in aggregate through observability, it's hard to gauge that any single org is having exactly repeatable results at they should expect vs. statistically acceptable volume for everyone. Because of this, we also setup synthetic accounts for test customers and regularly drive test scenarios through them to make sure the single customer doing the same old bring workloads are also doing alright. It tends to catch large issues that are caused changes that affect outputs without changing the volumes/latency. It's like end to end testing very common hot paths running forever in a real customer account flagged without billing. It tends to catch regressions way more often than it rightly should've.