Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

91–100 of 357 posts

Re: We don’t use a staging environment

#91

Post by rileymat2 »

jiggawatts wrote: post #64
rileymat2 wrote: post #44

Earlier quoted context omitted.

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

Non-prod is free.

The developer version, yes. But I have not seen the AWS amis for the developer version: https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-ec...

You can't install the enterprise non-prod for free. (But the developer version is supposed to have all the features)

Re: We don’t use a staging environment

#92

Post by robbiemitchell »

issa wrote: post #77

I have a lot of questions, but one above all the others. How do you preview changes to non-technical stakeholders in the company? Do you make sales people and CEOs and everyone else boot up a local development environment?

Also my main thought. Among other things, we sometimes use UAT as the place for broad QA on UX behavior a member of eng or data might not think to test. For quickly developed features that don’t go through a more formal design process, we’ll also review copy and styling.

Re: We don’t use a staging environment

#93

Post by midrus »

Good monitoring, logs, metrics, feature flagging (allowing for opening a branch of code for a % of users), blue/green deployment (allowing a release to handle a % of the user's traffic) and good tooling for quick builds/releases/rollback, in my experience, are far better tools than intermediate staging environments.

I've had great success in the past with a custom feature flags system + Google's App Engine % based traffic shifting, where you can send just a small % of traffic to a new service, and rollback to your previous version quickly without even needing to redeploy.

Now, not having those tools as a minimum, and not having either staging environment is just reckless. No unit/integration/whatever tests are going to make me feel safe about a deploy.

Re: We don’t use a staging environment

#94

Post by EsotericAlgo »

DandyDev wrote: post #48
nunez wrote: post #34

Earlier quoted context omitted.

There's a difference between permanent staging environments that need maintenance and disposable "staging" environments that are literally a clone of what's on your laptop that you trash once UAT/smoke is done. The former costs money and can lie to you; the latter is literally prod, but smaller.

This makes it sound so easy, but in my experience, permanent staging environments exist because setting up disposable staging environments is too complex. How do you deal with setting up complex infrastructure for your disposable staging environment when your system is more complex than a monolithic backend, some frontend and a (small) database? If your system consists of multiple components with complex interactions…

Absolutely. The answer is better integration boundaries but then you’re paying the abstraction cost which might be higher.

It’s particularly difficult when the system under test includes an application that isn’t designed to be set up ephemerally such as application-level managed services with only ClickOps configuration, proprietary systems where such a request is atypical and prevented by egregious licensing costs, or those that contain a physical component (e.g. a POS with physical peripherals).

Re: We don’t use a staging environment

#95

Post by midrus »

midrus wrote: post #93

Good monitoring, logs, metrics, feature flagging (allowing for opening a branch of code for a % of users), blue/green deployment (allowing a release to handle a % of the user's traffic) and good tooling for quick builds/releases/rollback, in my experience, are far better tools than intermediate staging environments. I've had great success in the past with a custom feature flags system + Google's App Engine % based tr…

And yes, you need blue/green deployments in addition to feature flags, as it is not easy to feature flag certain things, such as a language runtime version update or a third party library upgrade, among many other things.

Re: We don’t use a staging environment

#96

Post by drewcoo »

WYepQ4dNnG wrote: post #17

I don't see how this can scale beyond a single service. Complex systems are made of several services and infrastructure all interconnected. Things that are impossible to run on local. And even if you can run on local, the setup is most likely very different from production. The fact that things work on local give a little to zero guarantees that they will work in prod. If you have a fully automated infrastructure set…

> Complex systems are made of several services and infrastructure all interconnected.

Then maybe it's a forcing function to drive decoupling that tangle of code. That's a good thing!

Re: We don’t use a staging environment

#98

Post by drewcoo »

zimbatm wrote: post #70

If you can, provide on-demand environments for PRs. It's mostly helpful to test frontend changes, but also database migrations and just demoing changes to colleagues. If you have that, you will see people's behaviour change. We have a CTO that creates "demo" PRs with features they want to show to customers. All all the contension around staging as identified in the article is mostly gone.

You point out another kind of use of staging I've seen. "Don't touch staging until tomorrow after because SoAndSo is giving a demo to What'sTheirFace" so a bunch of engineering activity gets backed up.

Re: We don’t use a staging environment

#99

Post by cmeacham98 »

drewcoo wrote: post #85
kgeist wrote: post #73

Earlier quoted context omitted.

>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables…

> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the…

I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.

Re: We don’t use a staging environment

#100

Post by midrus »

lapser wrote: post #66

Disclaimer: I worked for a major feature flagging company, but these opinions are my own. This article makes a lot of valid points regarding staging environments, but their reasoning to not use them is dubious. None of their reasons are good enough to take staging environments out of the equation. I'd be willing to be that the likelihood of anyone merging code that isn't ready to go live is close to zero. You still n…

> If you truly want to get rid of your a staging environment the minimum that you need to feature flagging of _everything_, and I do mean everything. That is honestly near impossible. You also need live preview environments for each PR/branch. This somewhat eliminates the need for a staging because reviewers can test the changes on a live environment. These two things still aren't good enough reason to get rid of your staging environment. There is still many things that can go wrong.

This can be done very easily with many modern PaaS services. I had this like 6 or 7 years ago with Google App Engine, and we didn't have staging environment as each branch would be deployed and tested as if it were its own environment.

Post reply on HN