Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you test in production?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Ask HN: Do you test in production?

#61

Post by FoomFries »

logicalmonster wrote: post #60

Just to play Devil's Advocate and be argumentative, what is the point of testing in production when your development/staging environment is guaranteed to be identical to your production environment?

Is it? I would argue if you have live data in some large amount, you’re not going to be keeping that in sync with your dev environment (and it should be sanitized, which has its own fun oddities at times). I would also argue that your staging environment probably isn’t running the same hardware specs as live. Inevitably you’ll find race conditions, odd data structures and other fun things which simply aren’t apparent on systems prior to live.

Re: Ask HN: Do you test in production?

#62

Post by throw1138 »

A significant source of frustration at $dayjob recently has been the _inability_ to test in production. We've just deployed Stripe, and if you're using prod API keys, there's no testing possible without spending real money. Deploy to production and pray to the tech gods I guess.

Re: Ask HN: Do you test in production?

#63

Post by yellowapple »

One approach I recall taking as the in-house developer for a company's warehouse management system was to designate the warehouse I was in to be the "experimental" warehouse: new features (or new systems entirely!) would be developed/configured in coordination with that warehouse's team, and they were generally comfortable with the idea that they would get those new features first (with the risks that entails). Once my local site had used the new feature/system for a few weeks without major incident, it would then get rolled out to the other sites.

Re: Ask HN: Do you test in production?

#64

Post by paulryanrogers »

I tried this for a while, marking such tests as production compatible. They relied on test records made for the purpose, sometimes copied to other environments to make the tests.

For 3rd parties with test modes like Stripe you can get E2E, or if the cost of the test is low.

Some safety controls to avoid running non-prod-safe tests are wise.

Another alternative is using anonymized prod copies outside prod. Possibly even mocking 3rd parties to behave like prod, happy, sad, etc.

Re: Ask HN: Do you test in production?

#67

Post by brianwawok »

kasey_junk wrote: post #56
brianwawok wrote: post #3

Anytime you need to talk to a third party API, you need to test in prod. Some people have sandbox apis. They are generally broken and not worth it. See eBay for super in depth sandbox API that never works. You can read the docs 100 times over. At the end of the day, the API is going to work like it works. So you kind of “have to” test in prod for these guys.

cough CME cough

Hah did we even pretend to have a sandbox at CME? Outside like Saturday testing parties.

Re: Ask HN: Do you test in production?

#68

Post by pgwhalen »

logicalmonster wrote: post #60

Just to play Devil's Advocate and be argumentative, what is the point of testing in production when your development/staging environment is guaranteed to be identical to your production environment?

Another commenter made a good point that in many domains/systems, environment parity is by definition impossible because in production the system spends (non-trivial amounts of) money.

Re: Ask HN: Do you test in production?

#69

Post by jvans »

logicalmonster wrote: post #60

Just to play Devil's Advocate and be argumentative, what is the point of testing in production when your development/staging environment is guaranteed to be identical to your production environment?

In large systems an enormous amount of energy and dev time is wasted in trying to maintain staging/dev environments. Accepting that they won't ever be the same and building out tooling for safe testing/rollouts in prod is a far better use of resources

Re: Ask HN: Do you test in production?

#70

Post by kasey_junk »

brianwawok wrote: post #67
kasey_junk wrote: post #56

Earlier quoted context omitted.

cough CME cough

Hah did we even pretend to have a sandbox at CME? Outside like Saturday testing parties.

There was a sandbox but it never behaved like the prod environments. To the point I had to have a flag to tell my order entry code if we were being certified in the sandbox or actually trading in prod.

I’ve heard that got better after they replaced the gateways with fpga but don’t know for sure.

Post reply on HN