Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

71–80 of 357 posts

Re: We don’t use a staging environment

#71

Post by clintonb »

abhishekjha wrote: post #14
rock_hard wrote: post #8

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

That would be controlling a lot of feature flags given how many can be switched on at once. How do you control them?

I work at a different company. Typically feature flags are short-lived (on the order of days or weeks), and only control one feature. When I deploy, I only care about my one feature flag because that is the only thing gating the new functionality being deployed.

There may be other feature flags, owned by other teams, but it's rare to have flags that cross team/service boundaries in a fashion that they need to be coordinated for rollout.

Re: We don’t use a staging environment

#73

Post by kgeist »

rock_hard wrote: post #8

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

>That said features and bug fixes were often times gated by feature flags

Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables a feature toggle without running the required migrations, which can result in all sorts of downtime.

Another concern I have is that if a feature toggle isn't enabled in production for a long time (for us, several days is already a long time due to a tight release schedule) new changes to the codebase by another team can conflict with the disabled feature and, since it's disabled, you probably won't know there's a problem until it's too late?

Re: We don’t use a staging environment

#74

Post by lupire »

fishtoaster wrote: post #12

This is a pretty weird article. Their "how we do it" section lists: - "We only merge code that is ready to go live" - "We have a flat branching strategy" - "High risk features are always feature flagged" - "Hands-on deployments" (which, from their description, seems to be just a weird way of saying "we have good monitoring and observability tooling") ...absolutely none of which conflict with or replace having a stagi…

The company does some analysitics on highly redundant data (user behavior on website). They run a system with low requirements for a avaibility, correctness, and feature churn. Their product is nice to have but not important to mission on a daily basis. If their entire system went down for a day, or even 3 days a week, their customers would be only mildly inconvenienced. They aren't Amazon or Google. So they test in prod.

Re: We don’t use a staging environment

#75

Post by clintonb »

parksy wrote: post #53

This sounds like something I would write if a hypothetical gun was pointed at my head in a company where the most prominent customer complaint was that time spent in QA and testing was too expensive. I have zero trust in any company that deploys directly from a developer's laptop to production, not in the least starting with how much do you trust that developer. There has to be some process right?

My assumption is the process is more like this:

Laptop --> pull request + CI --> merge + CI + CD --> production

I don't think folks are pushing code directly via Git or SFTP.

Re: We don’t use a staging environment

#76

Post by drewcoo »

productceo wrote: post #26

> We only merge code that is ready to go live. In their perception, is the rest of tech industry gambling in every pull request that some untested code would work in production? I work at a large company. We extensively test code on local machines. Then dev test environments. Then small roll out to just a few data centers in prod bed. Run small scale online flight experiments. Then roll out to the rest of prod bed. A…

> I've seen code fail in each of the stages

How many of the failures caught in dev would have been legitimate problems in production? How about the ones in staging?

If your environments are that different are you even testing the right things?

And if yes, if you need all of those, then why not add a couple more environments? Because more pre-prod environments means more bugs caught in those, right? /s

Re: We don’t use a staging environment

#77

Post by issa »

I have a lot of questions, but one above all the others. How do you preview changes to non-technical stakeholders in the company? Do you make sales people and CEOs and everyone else boot up a local development environment?

Re: We don’t use a staging environment

#78

Post by bradleyjg »

lapser wrote: post #66

Disclaimer: I worked for a major feature flagging company, but these opinions are my own. This article makes a lot of valid points regarding staging environments, but their reasoning to not use them is dubious. None of their reasons are good enough to take staging environments out of the equation. I'd be willing to be that the likelihood of anyone merging code that isn't ready to go live is close to zero. You still n…

How do you feature flag a refactor?

Re: We don’t use a staging environment

#79

Post by user3939382 »

crummy wrote: post #55
user3939382 wrote: post #11

> Pre-live environments are never at parity with production Then you fix that particular problem. Infrastructure as code is one idea just off the top of my head.

Is it possible to make staging 100% identical with prod? Load is one thing I can think of that is difficult to make identical; even if you artificially generate it, user behaviour will likely be different.

I don't work on systems where that factor is critical to our tests, but if I was I would start here (at least in my case since we use AWS) https://docs.aws.amazon.com/solutions/latest/distributed-loa...

Re: We don’t use a staging environment

#80

Post by detaro »

bradleyjg wrote: post #78
lapser wrote: post #66

Disclaimer: I worked for a major feature flagging company, but these opinions are my own. This article makes a lot of valid points regarding staging environments, but their reasoning to not use them is dubious. None of their reasons are good enough to take staging environments out of the equation. I'd be willing to be that the likelihood of anyone merging code that isn't ready to go live is close to zero. You still n…

How do you feature flag a refactor?

You copy your service into refactored_service and feature-flag which of the two microservices the rest of the system uses /s
Post reply on HN