Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

41–50 of 357 posts

Re: We don’t use a staging environment

#41

Post by jeffbee »

What I infer from the article is this company does not handle sensitive private data, or they do but are unaware of it, or they are aware of it and just handle it sloppily. I infer that because one of the biggest advantages of a pre-prod environment is you can let your devs play around in a quasi-production environment that gets real traffic, but no traffic from outside customers. This is helpful because when you take privacy seriously there is no way for devs to just look at the production database, or to gain interactive shells in prod, or to attach debuggers to production services without invoke glass-breaking emergency procedures. In the pre-prod environment they can do whatever they want.

Most of the rest of the article is not about the disadvantages of pre-prod, but the drawbacks of the "git flow" branching model compared to "trunk based development". The latter is clearly superior and I agree with those parts of the article.

Re: We don’t use a staging environment

#42

Post by krm01 »

This isn’t very uncommon. In fact, it actually is exactly what the article is trying to explain it’s not: a staging/pre-live environment. Only instead of having it be deployed online, you keep it local.

Re: We don’t use a staging environment

#43

Post by _3u10 »

nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Having staging always encourages this. It’s really difficult to replicate prod in any non trivial way that exceeds what can be created on a workstation.

Eg. Even if you buy the same hardware you can’t replicate production load anyway because it’s not being used by 5 million people concurrently. Your cache access patterns aren’t the same, etc.

It’s far better to have a fast path to prod than a staging environment in my opinion.

Re: We don’t use a staging environment

#44

Post by rileymat2 »

nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

Re: We don’t use a staging environment

#45

Post by withinboredom »

sillysaurusx wrote: post #16
abhishekjha wrote: post #14

Earlier quoted context omitted.

That would be controlling a lot of feature flags given how many can be switched on at once. How do you control them?

flag = true More seriously, at my old company they just never got removed. So it wasn’t really about control. You just forgot about the ones that didn’t matter after awhile. If that sounds horrible, that’s probably the correct reaction. But it’s also common. Namespacing helps too. It’s easier to forget a bunch of flags when they all start with foofeature-.

I’ve seen those old flags come in handy once. Someone accidentally deleted a production database (typo) and we needed to stop all writes to restore from a backup. For most of it, it was just turning off the original feature flag, even though the feature was several years old.

Re: We don’t use a staging environment

#47

Post by skybrian »

sillysaurusx wrote: post #16
abhishekjha wrote: post #14

Earlier quoted context omitted.

That would be controlling a lot of feature flags given how many can be switched on at once. How do you control them?

flag = true More seriously, at my old company they just never got removed. So it wasn’t really about control. You just forgot about the ones that didn’t matter after awhile. If that sounds horrible, that’s probably the correct reaction. But it’s also common. Namespacing helps too. It’s easier to forget a bunch of flags when they all start with foofeature-.

It can become a code maintenance issue, though, when you revisit the code. You need to maintain both paths when you never know if they are being used.

Also, where flags interact, you can get a combinatorial explosion of cases to consider.

Re: We don’t use a staging environment

#48

Post by DandyDev »

nunez wrote: post #34
fishtoaster wrote: post #12

This is a pretty weird article. Their "how we do it" section lists: - "We only merge code that is ready to go live" - "We have a flat branching strategy" - "High risk features are always feature flagged" - "Hands-on deployments" (which, from their description, seems to be just a weird way of saying "we have good monitoring and observability tooling") ...absolutely none of which conflict with or replace having a stagi…

There's a difference between permanent staging environments that need maintenance and disposable "staging" environments that are literally a clone of what's on your laptop that you trash once UAT/smoke is done. The former costs money and can lie to you; the latter is literally prod, but smaller.

This makes it sound so easy, but in my experience, permanent staging environments exist because setting up disposable staging environments is too complex.

How do you deal with setting up complex infrastructure for your disposable staging environment when your system is more complex than a monolithic backend, some frontend and a (small) database? If your system consists of multiple components with complex interactions, and you can only meaningfully test features if there is enough data in the staging database and it's _the right_ data, then setting up disposable staging environments is not that easy.

Re: We don’t use a staging environment

#49

Post by mkl95 »

> People mistakenly let process replace accountability

> We only merge code that is ready to go live.

This is one of the most off-putting things I have read on HN lately. Having worked on several large SaaS where leadership claimed similar stuff, I simply refuse to believe it.

Re: We don’t use a staging environment

#50

Post by otterley »

MetaWhirledPeas wrote: post #36

I don't have experience with the true CI he describes, but I do have experience with pre-production environments. > "People mistakenly let process replace accountability" I find this to be mostly true. When the code goes somewhere else before it goes to prod, much of the burden of responsibility goes along with it. Other people find the bugs and spoon feed them back to the developers. I'm sure as a developer this is…

You can have both process and accountability. Process for the things that can be automated or subject to business rules; accountability for when the process fails (either by design or in its implementation) or after lapses in judgment.
Post reply on HN