Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

51–60 of 357 posts

Re: We don’t use a staging environment

#51

Post by tharkun__ »

fishtoaster wrote: post #12

This is a pretty weird article. Their "how we do it" section lists: - "We only merge code that is ready to go live" - "We have a flat branching strategy" - "High risk features are always feature flagged" - "Hands-on deployments" (which, from their description, seems to be just a weird way of saying "we have good monitoring and observability tooling") ...absolutely none of which conflict with or replace having a stagi…

FWIW I don't think it is weird at all. Maybe a little short on details of what ready really means for example. While I don't think going completely staging-less makes a lot of sense, going without a shared staging environment is a good thing.

It is absolutely awesome to be able to have your own "staging" environment for testing that is independent of everyone else. With the Cloud this is absolutely possible. Shared staging environments are really bad. Things that should take a day at most turn into a coordination and waiting game of weeks. And as pressure mounts to get things tested and out you might have people trying to deploy parts that "won't affect the other tests" going on at the same time. And then they do and you have no idea if it's your changes or their changes that made the tests fail. And since it's been 2 weeks since the change was made and you finally got time on that environment your devs have already finished working on two or more other changes in the meantime.

FWIW we have a similar set up where devs and QA can spin up a complete environment that is almost the exact same as prod and do so independently. They can turn on and off feature flags individually without affecting each other. Since we don't need to wait (except for the few minutes to deploy or a bit longer to create a new env from scratch) any bugs found can be fixed rather quickly as devs at most have started working on another task. The environment can be torn down once finished but probably will just be reused until the end of the day.

(while it's almost the same as prod it isn't completely like it for cost reasons meaning less nodes by default and such but honestly for most changes that is completely irrelevant and when it might be relevant it's easy to spin up more nodes temporarily through the exact same means as one would use to handle load spikes in prod).

Re: We don’t use a staging environment

#52

Post by kayodelycaon »

I don’t see how this works when you have multiple external services you don’t control in critical code paths that you can’t fully test in CI.

The cost of maintaining a staging environment is peanuts compared to 30 minutes of downtime or data corruption.

Re: We don’t use a staging environment

#53

Post by parksy »

This sounds like something I would write if a hypothetical gun was pointed at my head in a company where the most prominent customer complaint was that time spent in QA and testing was too expensive.

I have zero trust in any company that deploys directly from a developer's laptop to production, not in the least starting with how much do you trust that developer. There has to be some process right?

Re: We don’t use a staging environment

#54

Post by rio517 »

I struggle with a lot of the arguments made here. I think one key thing is that staging can mean different things. In the authors case, they say "can’t merge your code because someone else is testing code on staging." It is important to differentiate between this type of staging for development testing development branches vs a staging where only what's already merged for for deployment is automatically deployed.

Many of the problems are organizational/infrastructure challenges, not inherent to staging environments/setups. Straightening out dev processes and investing in the infrastructure solves most of the challenges discussed.

Their points:

What's wrong with staging environments?

* "Pre-live environments are never at parity with production" - resolved with proper investment in infrastructure.

* "There’s always a queue [for staging]" - is staging the only place to test pre-production code? If you need a place to test code that isn't in master, consider investing in disposable staging environments or better infrastructure so your team has more confidence for what they merge.

* "Releases are too large" - reduced queues reduces deployment times. Manage releases so they're smaller.

* "Poor ownership of changes" Of course this happens with all that queued code. address earlier challenges and this will be massively mitigated. Once there, good mangers's job is to ensure this doesn't happen.

* "People mistakenly let process replace accountability" - this is a management problem.

Solving some of the above challenges with the right investments creates a virtuous cycle of improvements.

How we ship changes at Squeaky?

* "We only merge code that is ready to go live" - This is quite arbitrary. How do you define/ensure this?

* "We have a flat branching strategy" - Great. It then surprises me that they have so much queued code and such large releases. I find it surprising they say, "We always roll forward." I wonder how this impacts their recovery time.

* "High risk features are always feature flagged" - do low risk features never cause problems?

* "Hands-on deployments" - I'm not sure this is good practice. How much focus does it take away from your team? Would a hands-off deployment with high confidence pre-deploy, automated deployment, automated monitoring and alerting, while ensuring the team is available to respond and recover quickly?

* "Allows a subset of users to receive traffic from the new services while we validate" is fantastic. Surprised they don't break this into its own thing.

Re: We don’t use a staging environment

#55

Post by crummy »

user3939382 wrote: post #11

> Pre-live environments are never at parity with production Then you fix that particular problem. Infrastructure as code is one idea just off the top of my head.

Is it possible to make staging 100% identical with prod? Load is one thing I can think of that is difficult to make identical; even if you artificially generate it, user behaviour will likely be different.

Re: We don’t use a staging environment

#56

Post by coder543 »

rileymat2 wrote: post #44
nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

If you're choosing to pay large sums of money for SQL Server instead of the open source alternatives, you should also factor in the large sums of money to have good development/staging environments too.

All the more reason to just use Postgres or MySQL.

EDIT: as someone else hinted at, it does look like the free Developer version of SQL Server is fully featured and licensed for use in any non-prod environment, which seems reasonable.

Re: We don’t use a staging environment

#57

Post by debarshri »

We used to believe staging environments are not important enough. If you believe that then I would argue that you have not crossed a threshold as an org where your product is critical enough for you consumers. The staging environment or any for that matter just acts as a gating mechanism to not ship crappy stuff to customers. You cannot have too many gates, then you would be shipping lates but with less number of gates you end up shipping low quality product.

Staging environment saves unnecessary midnight alerts and easy to catch issues that might have a huge impact when a customer has to face it. I wouldn't be surprised if in few quarters or a year or so they would have an article about why they decided to introduce a staging environment.

Re: We don’t use a staging environment

#58

Post by nickelpro »

_3u10 wrote: post #43
nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Having staging always encourages this. It’s really difficult to replicate prod in any non trivial way that exceeds what can be created on a workstation. Eg. Even if you buy the same hardware you can’t replicate production load anyway because it’s not being used by 5 million people concurrently. Your cache access patterns aren’t the same, etc. It’s far better to have a fast path to prod than a staging environment in m…

Perhaps we have different ideas about what a staging environment is for. I wouldn't expect a staging environment to give accurate performance numbers for a change, the only solution to that is instrumenting the production environment.

Re: We don’t use a staging environment

#59

Post by vasco »

DandyDev wrote: post #48
nunez wrote: post #34

Earlier quoted context omitted.

There's a difference between permanent staging environments that need maintenance and disposable "staging" environments that are literally a clone of what's on your laptop that you trash once UAT/smoke is done. The former costs money and can lie to you; the latter is literally prod, but smaller.

This makes it sound so easy, but in my experience, permanent staging environments exist because setting up disposable staging environments is too complex. How do you deal with setting up complex infrastructure for your disposable staging environment when your system is more complex than a monolithic backend, some frontend and a (small) database? If your system consists of multiple components with complex interactions…

It's not too complex. There's plenty of products that make this easy, gitlab review apps being one of them.

Re: We don’t use a staging environment

#60

Post by lolinder »

nunez wrote: post #34
fishtoaster wrote: post #12

This is a pretty weird article. Their "how we do it" section lists: - "We only merge code that is ready to go live" - "We have a flat branching strategy" - "High risk features are always feature flagged" - "Hands-on deployments" (which, from their description, seems to be just a weird way of saying "we have good monitoring and observability tooling") ...absolutely none of which conflict with or replace having a stagi…

There's a difference between permanent staging environments that need maintenance and disposable "staging" environments that are literally a clone of what's on your laptop that you trash once UAT/smoke is done. The former costs money and can lie to you; the latter is literally prod, but smaller.

Yeah, it sounds to me like OP had the former, which they've dropped, and haven't yet found a need for the latter.

I work for a tiny company that, when I joined, had a "pet" prod server and a "pet" staging server. The config between them varied in subtle but significant ways, since both had been running for 5 years.

I helped make the transition the article described and it was huge for our productivity. We went from releasing once a quarter to releasing multiple times a week. We used to plan on fixing bugs for weeks after a release, now they're rare.

We've since added staging back as a disposable system, but I understand where the author is coming from. "Pet" staging servers are nightmarish.

Post reply on HN