Note: if you plan on accurate financial planning and metrics (esp. if going public), you need to be able to separate your test prod stats from the real prod stats for reporting.
Ask HN: Do you test in production?
11–20 of 75 posts
Re: Ask HN: Do you test in production?
#12A/B tests and feature flags are basically testing in prod. And yes, some of those features sometimes run as a "well, it should work, but we're not entirely sure until we get a significant number of users using the system". It could be an edge case failing or scalability requirements being wrong.
Another variation on the same theme is rewriting systems when you run production data through both systems. Quite often that's the only way of doin migrations to a new platform, or a new database, or yes, a newly re-written system.
> Is it common practice to inject fabricated data into a prod system to run such tests? What's the best practice or prior art on doing this well?
A very common practice is to run a snapshot of prod data (e.g. last hour, or last 24 hours, or even a week/month/year) through a system in staging (or cooking, or pre-cooking, or whatever name you give the system that's just about to be released). However, doing it properly may not be easy, and depends on the systems involved.
Re: Ask HN: Do you test in production?
#13Sometimes one cannot get the exact same specs on test hardware versus production, yet a rollout depends on simulating system load to shake out issues.
Performance testing needs a schedule, visibility, timebox, known scope, backout plan, data revert plan, pre- and post-graphs.
Schedule. Folks are clearly tagged in a
table with times down the side.
Visibility. Folks who should know know
when it's going to happen,
are invited to the session,
and are mentioned in the
distributed schedule.
Timebox. It's going to start at a
defined time and end on a
defined time.
Known scope. Is it going to fulfill an
order? How many accounts
created?
Backout plan. DBA and DevOps on standby
for stopping the test.
Data revert plan. We know what rows to
delete or update after
testing.
Pretty pictures. You want to show graphs
during the test, so
that you know what to
improve and everyone's
time wasn't wasted.
Re: Ask HN: Do you test in production?
#14Anytime you need to talk to a third party API, you need to test in prod. Some people have sandbox apis. They are generally broken and not worth it. See eBay for super in depth sandbox API that never works. You can read the docs 100 times over. At the end of the day, the API is going to work like it works. So you kind of “have to” test in prod for these guys.
Ditto regarding Paypal: you need a sandbox API token to get started with it. Their sandbox token generator was broken for MONTHS, I could not believe it. By the time we got the token, we already fixed all bugs on our side the hard way - by testing in prod - and moved on.
Re: Ask HN: Do you test in production?
#15I work for a B2E company that has a structure similar to Salesforce. We test in production all the time even for our secure environments where the data is highly sensitive. Re: data, it’s a somewhat common practice to notionalize data (think isomorphically faking data). We regularly do this and will often designate rows as notional to hide them from users who aren’t admins. I’ve found this to work exceptionally well;…
Thanks. This sounds interesting.
Can you give a bit more colour on "notionalizing" and "isomorphically faking" please.
Re: Ask HN: Do you test in production?
#16I work for a B2E company that has a structure similar to Salesforce. We test in production all the time even for our secure environments where the data is highly sensitive. Re: data, it’s a somewhat common practice to notionalize data (think isomorphically faking data). We regularly do this and will often designate rows as notional to hide them from users who aren’t admins. I’ve found this to work exceptionally well;…
>notionalize data (think isomorphically faking data)
Are these just $5 words for setting a fake data flag on the records?
Re: Ask HN: Do you test in production?
#17yes you can do so with a canary tier . assuming your code is well instrumented to distinguish performance and quality regressions , a canary tier served to customers will catch more regressions than synthetic testing
Re: Ask HN: Do you test in production?
#18I work for a B2E company that has a structure similar to Salesforce. We test in production all the time even for our secure environments where the data is highly sensitive. Re: data, it’s a somewhat common practice to notionalize data (think isomorphically faking data). We regularly do this and will often designate rows as notional to hide them from users who aren’t admins. I’ve found this to work exceptionally well;…
>notionalize data (think isomorphically faking data) Are these just $5 words for setting a fake data flag on the records?
We do that too but notionalizing for us is usually creating data that looks and behaves realistically but is actually fake. (A side benefit to this is that we can then use it for demos!)
Re: Ask HN: Do you test in production?
#19Depends a lot on your application and how big the changes are. IF you're an online store and you're pushing out incremental changes to a subset of users its a good strategy. If its aircraft auto-pilot not so much.
Re: Ask HN: Do you test in production?
#20Earlier quoted context omitted.
>notionalize data (think isomorphically faking data) Are these just $5 words for setting a fake data flag on the records?
We do that too but notionalizing for us is usually creating data that looks and behaves realistically but is actually fake. (A side benefit to this is that we can then use it for demos!)
So you mock data and then flag it as fake.