Live data from Hacker News

We don’t use a staging environment

squeaky.ai

31–40 of 357 posts

Re: We don’t use a staging environment

#31
This makes sense. With a high-enough release velocity to trunk, a super safe release pipeline with lots of automated checks, a well-tested rolling update/rollback process in production, and aggressive observability, it is totally possible to remove staging in many environments. This is one of the popular talking points touted by advocates of trunk-based development.

(Note that you can do a lot of exploratory testing in disposable environments that get spun up during CI. Since the code in prod is the same as the code in main, there's no reason to keep them around. That's probably how they get around what's traditionally called UAT.)

The problem for larger companies that tend to have lots of staging environments is that the risk of testing in production vastly exceeds the benefits gained from this approach. Between the learning curve required to make this happen, the investment required to get people off of dev, the significantly larger amounts of money at stake, and, in many cases, stockholder responsibilities, it is an uphill battle to get companies to this point.

Also, many (MANY) development teams at BigCo's don't even "own" their code once it leaves staging.

I've found it easier to employ a more grassroots approach towards moving people towards laptop-to-production. Every dev wants to work like Squeaky does (many hate dev/staging environments for the reasons they've outlined); they just don't feel empowered to do so. Work with a single team that ships something important but won't blow up the company if they push a bad build into prod. Let them be advocates internally to promote (hopefully) pseudo-viral spread.

Re: We don’t use a staging environment

#32
At my previous job we had a single staging environment, which was used by dozens of teams to test independent releases as well as to test our public mobile app before release. That said, it never matched production, so releases were always a crapshoot as things suddenly happened no one ever tested. Yes, it was dumb.

Re: We don’t use a staging environment

#34

This is a pretty weird article. Their "how we do it" section lists: - "We only merge code that is ready to go live" - "We have a flat branching strategy" - "High risk features are always feature flagged" - "Hands-on deployments" (which, from their description, seems to be just a weird way of saying "we have good monitoring and observability tooling") ...absolutely none of which conflict with or replace having a stagi…

There's a difference between permanent staging environments that need maintenance and disposable "staging" environments that are literally a clone of what's on your laptop that you trash once UAT/smoke is done.

The former costs money and can lie to you; the latter is literally prod, but smaller.

Re: We don’t use a staging environment

#35
This is how we work at fastcomments... soon we will have a shard in each major continent and will just deploy changes to a shard, run e2e tests, and then roll out to the rest of the Shards.

But if you have a high risk system or a business that values absolute quality over iteration speed, then yeah you want dev/staging envs...

Re: We don’t use a staging environment

#36
I don't have experience with the true CI he describes, but I do have experience with pre-production environments.

> "People mistakenly let process replace accountability"

I find this to be mostly true. When the code goes somewhere else before it goes to prod, much of the burden of responsibility goes along with it. Other people find the bugs and spoon feed them back to the developers. I'm sure as a developer this is nice, but as a process I hate it.

Re: We don’t use a staging environment

#37
The short answer appears to be "we are cheap and nobody cares yet."

It's easy to damn the torpedoes and deploy straight into production if there's nobody to care about, or your paying customers (to the extent you have any) don't care either.

Once you start gaining paying customers who really care about your service being reliable, your tune changes pretty quickly. If your customers rely on data fidelity, they're going to get pretty steamed when your deployment irreversibly alters or irrevocably loses it.

Also, "staging never looks like production" looks like a cost that tradeoff that the author made, not a Fundamental Law of DevOps. If you want it to look like production, you can do the work and develop the discipline to make it so. The cloud makes this easier than ever, if you're willing to pay for it.

Re: We don’t use a staging environment

#39

> We only merge code that is ready to go live. In their perception, is the rest of tech industry gambling in every pull request that some untested code would work in production? I work at a large company. We extensively test code on local machines. Then dev test environments. Then small roll out to just a few data centers in prod bed. Run small scale online flight experiments. Then roll out to the rest of prod bed. A…

Yeah, it seems like someone took RFC 9225 to heart. (https://www.rfc-editor.org/rfc/rfc9225.html)
Post reply on HN