Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

151–160 of 357 posts

Re: We don’t use a staging environment

#151

Post by fmakunbound »

I’m working at megacorp at the moment as contractor. The local dev, cloud dev, cloud stage, cloud prod pipeline is truly glacial in velocity even with automation like Jenkins, kubernetes, etc. it takes weeks to move from dev complete to production. It’s a middle manager’s wet dream.

I used to wonder why isn’t megacorp being murdered by competitors delivering features faster, but actually, everyone is moving glacially for the same reason, so it doesn’t matter.

I’m kinda reminded by pg’s essay on which competitors to worry about. I might be a worried competitor if these guys are pulling off merging to master as production.

Re: We don’t use a staging environment

#152

Post by tilolebo »

drewcoo wrote: post #111
debarshri wrote: post #57

We used to believe staging environments are not important enough. If you believe that then I would argue that you have not crossed a threshold as an org where your product is critical enough for you consumers. The staging environment or any for that matter just acts as a gating mechanism to not ship crappy stuff to customers. You cannot have too many gates, then you would be shipping lates but with less number of gat…

This reminds me of the "bake time" arguments I've had. There's some magical idea that if software "bakes" in an environment for some unknowable amount of time, it will be done and ready to deploy. Very superstitious. what is the actual value gained from staging specifically? Once you have a list of those, a specific list, figure out why only staging could do that and not testing before or after. And "it's caught bugs…

> And "it's caught bugs before" is not good enough.

Why isn't it good enough?

Re: We don’t use a staging environment

#153

Post by [deleted] »

tedmiston wrote: post #144
jasonhansel wrote: post #137

This is good insofar as it forces you to make local development possible. In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally. That leads to people only testing in staging envs, causing staging to constantly break and discouraging automated tests that prevent regression bugs. It also leads to increasing complexity and interconn…

Ehh... once your systems use more than a few pieces of cloud infrastructure / SaaS / PaaS / external dependencies / etc, purely local development of the system is just not possible. There are some (limited) simulators / emulators / etc available and whatnot for some services, but running a full platform that has cloud dependencies on a local machine is often just not possible.

[deleted]

Re: We don’t use a staging environment

#154

Post by paulryanrogers »

jasonhansel wrote: post #142
Negitivefrags wrote: post #125

If you are saying you don't have a staging environment, what you are really saying is that your company doesn't have any QA process. If your QA process is just developers testing their own shit on their local machine then you are not going to get as much value out of staging.

But you don't need to have a single staging env shared by all QA testers. Why not create individual QA environments on an as-needed basis for testing specific features? Of course this requires you to invest in making it easy to create new environments, but it allows QA teams to test different things without interfering with each other.

This worked reasonably well as v-hosts per engineer, though it did share some production resources. QA members would then run through test plans against those hosts to exercise the code. I prefer it to a single monolithic env. Though branches had to be kept up to date and bigger features tested as whole.

Re: We don’t use a staging environment

#155

Post by debarshri »

drewcoo wrote: post #111
debarshri wrote: post #57

We used to believe staging environments are not important enough. If you believe that then I would argue that you have not crossed a threshold as an org where your product is critical enough for you consumers. The staging environment or any for that matter just acts as a gating mechanism to not ship crappy stuff to customers. You cannot have too many gates, then you would be shipping lates but with less number of gat…

This reminds me of the "bake time" arguments I've had. There's some magical idea that if software "bakes" in an environment for some unknowable amount of time, it will be done and ready to deploy. Very superstitious. what is the actual value gained from staging specifically? Once you have a list of those, a specific list, figure out why only staging could do that and not testing before or after. And "it's caught bugs…

Firstly, There is no magical idea of software "baking" in an environment. It is about the risk appetite of the org., how willing is an org to push a feature that is "half-baked" their customers.

I believe modern day testing infrastructure looks very different. I have seen products like ReleaseHub that provides ondemand environments to dev to testing their changes out which eliminates the need for common testing env. That naturally means you need atleast one "pre-release" environment where all the changes are which would eventually becomes the next release. If you don't have this "pre-release" environment you will never be able to capture the side-effects of all the parallel changes that are happening to the codebase.

Thirdly, you have to see the context. When you have a microservice architecture, having a staging environment does not matter as fault tolerance, circuit breaking and other concepts makes sure that failed deployment of one services does not impact others. However, when you have a monolithic architecture you will never know what the side-effects of changes are unless you have a staging environment which would get promoted to production.

If you value customers, you should have a staging environment as a guardrail. The cost of not adhering or having a process like this is huge and possibly company-ending.

Re: We don’t use a staging environment

#156

Post by bob1029 »

> Pre-live environments are never at parity with production

As a B2B vendor, this is a conclusion we have been forced to reach across the board. We have since learned how to convince our customers to test in production.

Testing in prod is usually really easy if you are willing to have a conversation with the other non-technical humans in the business. Simple measures like a restricted prod test group are about 80% of the solution for us.

Re: We don’t use a staging environment

#157

Post by marvinblum »

I use a somewhat similar approach for Pirsch [0]. It's build so that I can run it locally, basically as a fully fledged staging environment. Databases run in Docker, everything else is started using modd [1]. This has proven to be a good setup for quick iterations and testing. I can quickly run all tests on my laptop (Go and TypeScript) and even import data from production to see if the statistics are correct for real data. Of course, there are some things that need to be mocked, like automated backups, but so far it turned out to work really well.

You can find more on our blog [2] if you would like to know more.

[0] https://pirsch.io

[1] https://github.com/cortesi/modd

[2] https://pirsch.io/blog/techstack/

Re: We don’t use a staging environment

#158

Post by morelisp »

Negitivefrags wrote: post #125

If you are saying you don't have a staging environment, what you are really saying is that your company doesn't have any QA process. If your QA process is just developers testing their own shit on their local machine then you are not going to get as much value out of staging.

No, it says if you have a QA process it doesn't including a staging environment.

A QA process is just a process - it doesn't have necessary parts - as long as it's finding the right balance between cost, velocity, and risk for your needs, it's working. Some parts like CI are nearly universal now that they're so cheap; some like feature flags managed in a distributed control plane are expensive; some like staging deployments are somewhere in the middle.

Re: We don’t use a staging environment

#159

Post by [deleted] »

Negitivefrags wrote: post #125

If you are saying you don't have a staging environment, what you are really saying is that your company doesn't have any QA process. If your QA process is just developers testing their own shit on their local machine then you are not going to get as much value out of staging.

[deleted]

Re: We don’t use a staging environment

#160

Post by bob1029 »

rileymat2 wrote: post #44
nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

> Depending on your tech, staging environments can be very expensive

For our business & customers, a new staging environment means another phone call to IBM and a ~6 month wait before someone even begins to talk about how much money its gonna cost.

Post reply on HN