Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

61–70 of 357 posts

Re: We don’t use a staging environment

#61

Post by rileymat2 »

coder543 wrote: post #56
rileymat2 wrote: post #44

Earlier quoted context omitted.

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

If you're choosing to pay large sums of money for SQL Server instead of the open source alternatives, you should also factor in the large sums of money to have good development/staging environments too. All the more reason to just use Postgres or MySQL. EDIT: as someone else hinted at, it does look like the free Developer version of SQL Server is fully featured and licensed for use in any non-prod environment, which…

Sure different planning 20 years ago would have made a big difference. Or the will/resources to transition. I am just saying that this scenario exists.

Re: We don’t use a staging environment

#62

Post by mdoms »

> "We only merge code that is ready to go live"

I like to go even farther, I advocate only merging code that won't break anything. If you're feature flagging as many changes as possible then you can merge code that doesn't even work, as long as you can gate users away from it using feature flags. The sooner and more often you can integrate unfinished code (safely) into master the better.

Re: We don’t use a staging environment

#63

Post by tharkun__ »

DandyDev wrote: post #48
nunez wrote: post #34

Earlier quoted context omitted.

There's a difference between permanent staging environments that need maintenance and disposable "staging" environments that are literally a clone of what's on your laptop that you trash once UAT/smoke is done. The former costs money and can lie to you; the latter is literally prod, but smaller.

This makes it sound so easy, but in my experience, permanent staging environments exist because setting up disposable staging environments is too complex. How do you deal with setting up complex infrastructure for your disposable staging environment when your system is more complex than a monolithic backend, some frontend and a (small) database? If your system consists of multiple components with complex interactions…

Sibling here but I can talk a bit about how we do it.

Through infrastructure as code. We do not have a monolithic backend. We have a bunch of services, some smaller, some bigger. Yes there's "some frontend" but it's not just one frontend. We have multiple different "frontend services" serving different parts of it. As for database, we use multiple different database technologies, depending on the service. Some service uses only one of those, while others use a mix that is suited best to a particular use case. For one of those we use sharding and while a staging or dev environment doesn't need the sharding, these obviously use the only shard we create in dev/staging but the same mechanism for shard lookup are used. For data it depends. We have a data generator that can be loaded with different scenarios, either generator parameters or full fledged "db backup style" definitions that you can use but don't have to. We deploy to Prod multiple times per day (basically relatively shortly after something hits the main branch).

Through the exact same means we could also re-create prod at any time and in fact DR exercises are held for that regularly.

Re: We don’t use a staging environment

#64

Post by jiggawatts »

rileymat2 wrote: post #44
nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

Non-prod is free.

Re: We don’t use a staging environment

#66

Post by lapser »

Disclaimer: I worked for a major feature flagging company, but these opinions are my own.

This article makes a lot of valid points regarding staging environments, but their reasoning to not use them is dubious. None of their reasons are good enough to take staging environments out of the equation.

I'd be willing to be that the likelihood of anyone merging code that isn't ready to go live is close to zero. You still need to validate the code. Their branching strategy is (in my opinion) the ideal branching strategy, but again, that isn't good enough to take staging away.

Using feature flags is probably the only reason they give that comes to close to being okay with getting rid of staging, but even then, you can't always be sure that the code you've built works as expected. So you still need a staging environment to validate some things.

Having hands-on deployments should always be happening anyway. It's not a reason to not have a staging environment.

If you truly want to get rid of your a staging environment the minimum that you need to feature flagging of _everything_, and I do mean everything. That is honestly near impossible. You also need live preview environments for each PR/branch. This somewhat eliminates the need for a staging because reviewers can test the changes on a live environment. These two things still aren't good enough reason to get rid of your staging environment. There is still many things that can go wrong.

The reason we have layered deployment systems (CI, staging etc) is to increase confidence that your deployment will be good. You can never be 100% sure. But I'll bet you, removing a staging environment lowers that confidence further.

Having said all of this, if it works for you, then great. But the reasons I've read on this post, don't feel good enough to me to get rid of any staging environments.

Re: We don’t use a staging environment

#67

Post by mdoms »

sillysaurusx wrote: post #16
abhishekjha wrote: post #14

Earlier quoted context omitted.

That would be controlling a lot of feature flags given how many can be switched on at once. How do you control them?

flag = true More seriously, at my old company they just never got removed. So it wasn’t really about control. You just forgot about the ones that didn’t matter after awhile. If that sounds horrible, that’s probably the correct reaction. But it’s also common. Namespacing helps too. It’s easier to forget a bunch of flags when they all start with foofeature-.

At a previous workplace we managed flags with Launch Darkly. We asked developers not to create flags in LD directly but used Jira web hooks to generate flags from any Jira issues of type Feature Flag. This issue type had a workflow that ensured you couldn't close off an epic without having rolled out and then removed every feature flag. Flags should not significantly outlast their 100% rollout.

Re: We don’t use a staging environment

#69

Post by drewcoo »

fishtoaster wrote: post #12

This is a pretty weird article. Their "how we do it" section lists: - "We only merge code that is ready to go live" - "We have a flat branching strategy" - "High risk features are always feature flagged" - "Hands-on deployments" (which, from their description, seems to be just a weird way of saying "we have good monitoring and observability tooling") ...absolutely none of which conflict with or replace having a stagi…

Those bullets together explain how they can avoid having a staging environment.

There's a whole section of the article entitled "What’s wrong with staging environments?" that explains why they don't want staging.

They even presented their "why" before going into their "how." There is absolutely nothing weird about this.

Well, ok, it's weird that not all so-called "software engineers" follow this pattern of problem-solving. But that's not Squeaky's fault. They're showing us how to do it better.

Re: We don’t use a staging environment

#70

Post by zimbatm »

If you can, provide on-demand environments for PRs. It's mostly helpful to test frontend changes, but also database migrations and just demoing changes to colleagues.

If you have that, you will see people's behaviour change. We have a CTO that creates "demo" PRs with features they want to show to customers. All all the contension around staging as identified in the article is mostly gone.

Post reply on HN