21–30 of 75 posts
Some people try to NOT test in production, but everyone does test in prod in a very real sense because dependencies and environments are different in prod.
I think the question was "Do you INTENTIONALLY test in production"
However flags turn on/off code, not data, and my main area of interest here is how to deal with the test data problem in prod.
I work for a B2E company that has a structure similar to Salesforce. We test in production all the time even for our secure environments where the data is highly sensitive. Re: data, it’s a somewhat common practice to notionalize data (think isomorphically faking data). We regularly do this and will often designate rows as notional to hide them from users who aren’t admins. I’ve found this to work exceptionally well;…
Thanks. This sounds interesting. Can you give a bit more colour on "notionalizing" and "isomorphically faking" please.
The goal here is for the data to both be useful for testing and provide coverage not just at a software level, but at a user story level. This helps test things like cross-application interactions; is also doubly helpful since we can use it for demos without screwing up production data.
Some basic ping tests and other checks before swapping (as in preparing, initiating, and pointing the load balancer) to a new version into production would be smart.
Earlier quoted context omitted.
We do that too but notionalizing for us is usually creating data that looks and behaves realistically but is actually fake. (A side benefit to this is that we can then use it for demos!)
So you mock data and then flag it as fake.
Just now realizing notionalizing isn’t a widely accepted term for this
Once you're into testing I/O, which is ultimately unavoidable no matter how hard you try not to, you either need cooperative third parties who can give you truly representative test systems (rare) or a certain amount of test-in-prod.
Testing database stuff remains hard. You either wrap things in a some kind of layer you can mock out, or dupe prod or some subset of it into a staging environment with a daily snapshot or similar and hope any differences (scale, normally) aren't too bad.
Copy-on-write systems or those with time-travel and/or immutability help immensely with test-in-prod, especially if you can effectively branch your data. If it's your own systems you are testing against, things like lakefs.io look pretty useful in this regard.
And yes, feature flags, good metrics, and load balancers that you can send a small percentage of traffic through a new version (if your traffic/system allows such things) all help.
A common dev concern (usually raised by people who have no idea how users actually use stuff!) is that someone might pick up the report and then do something awful based on it, which would be awful^2 - I then explain that users can't find/use the updated reports till we tell them where they are and grant access permissions etc etc, so it's going to be fine and there's no need to panic, which calms them down till they forget by the time this comes up again!
On a side note, these people seem to get much more wound up about principal based worries (it would be bad to test in Prod being a prime example) compared to concerns based on their own weaknesses (ie they rush, forget a whole section of requirements, make mistakes, can't spot obvious bugs) which they seem to imagine are way less likely to cause problems than experience demonstrates.
In almost every case you cannot test new functionality on actual prod data, at least not anything thats not strictly “read only”-functionality. If you have a new feature to send automated mail to someone foreclosing on their property you just do not test that on a real live system.
What you can do is setup a staging environment that is as close to prod config as possible, and then copy the prod database to the staging env. Do your tests in staging. It doesnt matter if data in stg is messed ud. There may well be legal or company policy or security restrictions preventing you from doing this, but its the only way to test on real life data without the risk of f**ing up data in the live system.
Then there are integration tests - to other systems that is, which are a much harder problem
A/B tests and feature flags are basically testing in prod. And yes, some of those features sometimes run as a "well, it should work, but we're not entirely sure until we get a significant number of users using the system". It could be an edge case failing or scalability requirements being wrong. Another variation on the same theme is rewriting systems when you run production data through both systems. Quite often tha…
This doesn't mean it's not possible to do, but it's a pretty big lift to get right, and keep right.