Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

111–120 of 357 posts

Re: We don’t use a staging environment

#111

Post by drewcoo »

debarshri wrote: post #57

We used to believe staging environments are not important enough. If you believe that then I would argue that you have not crossed a threshold as an org where your product is critical enough for you consumers. The staging environment or any for that matter just acts as a gating mechanism to not ship crappy stuff to customers. You cannot have too many gates, then you would be shipping lates but with less number of gat…

This reminds me of the "bake time" arguments I've had. There's some magical idea that if software "bakes" in an environment for some unknowable amount of time, it will be done and ready to deploy. Very superstitious.

what is the actual value gained from staging specifically? Once you have a list of those, a specific list, figure out why only staging could do that and not testing before or after. And "it's caught bugs before" is not good enough.

Re: We don’t use a staging environment

#112

Post by rileymat2 »

booi wrote: post #104
rileymat2 wrote: post #91

Earlier quoted context omitted.

The developer version, yes. But I have not seen the AWS amis for the developer version: https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-ec... You can't install the enterprise non-prod for free. (But the developer version is supposed to have all the features)

It's pretty easy to create your own amis with developer versions. It makes sense why AWS doesn't necessarily provide this out of the box. But it still stands for fully managed versions of licensed software, you'll pay for the license even if it's non-prod

Yes, that's not to say it is not possible to create a similar env, but I thought the debate was how precisely you are replicating your production env.

Sure it may be "good enough", but I thought the debate was about precision. How your own ami setup may differ from the AWS built from the developer version compared to the AWS ami? I don't know.

Trying for an identical setup in staging is expensive, this is just a scenario I am familiar with. I am sure there are a lot like this.

Re: We don’t use a staging environment

#113

Post by shoo »

drewcoo wrote: post #98
zimbatm wrote: post #70

If you can, provide on-demand environments for PRs. It's mostly helpful to test frontend changes, but also database migrations and just demoing changes to colleagues. If you have that, you will see people's behaviour change. We have a CTO that creates "demo" PRs with features they want to show to customers. All all the contension around staging as identified in the article is mostly gone.

You point out another kind of use of staging I've seen. "Don't touch staging until tomorrow after because SoAndSo is giving a demo to What'sTheirFace" so a bunch of engineering activity gets backed up.

in enterprisey environments with large numbers of integrated services, its even worse if a single staging environment is used to do end-to-end integration testing involving many systems. lots of resource contention for access to staging environment.

Re: We don’t use a staging environment

#114

Post by laurent123456 »

cmeacham98 wrote: post #99
drewcoo wrote: post #85

Earlier quoted context omitted.

> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the…

I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.

I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.

Re: We don’t use a staging environment

#115

Post by parksy »

drewcoo wrote: post #87
parksy wrote: post #53

This sounds like something I would write if a hypothetical gun was pointed at my head in a company where the most prominent customer complaint was that time spent in QA and testing was too expensive. I have zero trust in any company that deploys directly from a developer's laptop to production, not in the least starting with how much do you trust that developer. There has to be some process right?

> company that deploys directly from a developer's laptop to production Luckily, there's no sign of doing that here. There's no mention of how their CI/CD works, probably because it's out of scope for an already long article, but that's clearly happening.

"We only have two environments: our laptops, and production. Once we merge into the main branch, it will be immediately deployed to production."

Maybe my reading skills have completely vanished but to me, this exactly says they deploy directly from their developers' laptops to production. Those are literally the words used. The rest of the article goes on to defend not having a pre production environment.

They literally detail how they deploy from their laptops to production with no other environments and make arguments for why that's a good thing.

Re: We don’t use a staging environment

#116

Post by mr337 »

otterley wrote: post #37

The short answer appears to be "we are cheap and nobody cares yet." It's easy to damn the torpedoes and deploy straight into production if there's nobody to care about, or your paying customers (to the extent you have any) don't care either. Once you start gaining paying customers who really care about your service being reliable, your tune changes pretty quickly. If your customers rely on data fidelity, they're goin…

Ooof I think I have to agree with "we are cheap and nobody cares yet.". If we had a bad release go out that blocked nightly processing, for example, it was how amazing fast it became a ticket to CEOs start calling.

One of the things that we did really well is we had tooling that spun up environments. The same tooling DevOps stood up production environments also stood up environments for PRs and UAT. Anyone within the company could spin up an environment for which ever reason be it from master or to apply a PR. When it works it works great, if it doesn't work fix it and don't throw out the entire concept.

Re: We don’t use a staging environment

#117

Post by jurschreuder »

NorwegianDude wrote: post #108

Staging, tests, previews and even running code locally is for people who make mistakes. It's dumb and a total waste of time if you don't make any mistakes. No testing at all, that's what I call optimizing for success! On a more serious note: Sometimes staging is the same as local, and in those situations there is very limited use for staging.

We often deploy to production directly because a customer wants a feature right now. I was thinking of changing the staging server to be called beta. Customers can use new features directly, but at their own risk.

Re: We don’t use a staging environment

#118

Post by hetspookjee »

jurschreuder wrote: post #117
NorwegianDude wrote: post #108

Staging, tests, previews and even running code locally is for people who make mistakes. It's dumb and a total waste of time if you don't make any mistakes. No testing at all, that's what I call optimizing for success! On a more serious note: Sometimes staging is the same as local, and in those situations there is very limited use for staging.

We often deploy to production directly because a customer wants a feature right now. I was thinking of changing the staging server to be called beta. Customers can use new features directly, but at their own risk.

I’ve seen that before but then called acceptance with a select group.

Re: We don’t use a staging environment

#119

Post by myth2018 »

I'm assuming this is not an April Fools' joke, and my comments are targeted at the discussion it sparked here anyway.

A flat branching model simplify things, and the strategy they describe surely enables them to ship features to production faster. But the risks I see there:

- who decides when a feature is ready to go to production? The programmer who developed them? The automated tests?

- features toggleable by a flag must, at least ideally, be double-tested -- both when turned on and off. Being in a hurry to deploy to production wouldn't help on that;

- OK, staging environments aren't in parity with production. But wouldn't they be better than the CD/CI pipeline, or developer's laptop, testing new features in isolation?

- Talking about features in isolation: what about bugs caused by spurious interaction between two or more features? No amount of test would find them if they only test features in isolation

Re: We don’t use a staging environment

#120

Post by colonwqbang »

rileymat2 wrote: post #44
nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

You could call that an infrastructure problem. You have built an expensive infrastructure which you cannot afford to scale to the extent you desire.
Post reply on HN