Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

291–300 of 357 posts

Re: We don’t use a staging environment

#291

Post by aniforprez »

drewcoo wrote: post #76
productceo wrote: post #26

> We only merge code that is ready to go live. In their perception, is the rest of tech industry gambling in every pull request that some untested code would work in production? I work at a large company. We extensively test code on local machines. Then dev test environments. Then small roll out to just a few data centers in prod bed. Run small scale online flight experiments. Then roll out to the rest of prod bed. A…

> I've seen code fail in each of the stages How many of the failures caught in dev would have been legitimate problems in production? How about the ones in staging? If your environments are that different are you even testing the right things? And if yes, if you need all of those, then why not add a couple more environments? Because more pre-prod environments means more bugs caught in those, right? /s

What? What even is this

The whole point of a dev environment is so you can catch errors you wouldn't in production. Keeping it as close as possible to production is the point. If you're catching errors in dev, you'll see them in production

More environments don't catch more bugs. What is this corollary? More testing catches more bugs. A dev environment allows free integration testing without act users of production being affected

Re: We don’t use a staging environment

#293

Post by ygouzerh »

It might be enough for this company, but if you are a big corporate, it's definitely not something to do. You cannot expect millions of consumers to just be ok with the fact that the mobile app is done because it's too hard to keep in sync staging and prod.

I am maintaining the infra for a big mobile app and our staging environment allowed us in the last year to have only two production incidents and they were not due to code source (networking).

I really recommend any serious business to at least try it and see by themselves the advantages

Re: We don’t use a staging environment

#294

Post by nunez »

DandyDev wrote: post #48
nunez wrote: post #34

Earlier quoted context omitted.

There's a difference between permanent staging environments that need maintenance and disposable "staging" environments that are literally a clone of what's on your laptop that you trash once UAT/smoke is done. The former costs money and can lie to you; the latter is literally prod, but smaller.

This makes it sound so easy, but in my experience, permanent staging environments exist because setting up disposable staging environments is too complex. How do you deal with setting up complex infrastructure for your disposable staging environment when your system is more complex than a monolithic backend, some frontend and a (small) database? If your system consists of multiple components with complex interactions…

it's actually "pretty easy" to do when you start from first principles.

I usually ask "can I build your code on my laptop? is this the same as what's in prod?" usually the answer is no, so I work to turn that into a yes.

often times, I find that much of the complexity that you speak of is due to shared services that few have invested time into running locally precisely because of long-lived dev/staging envs, like access to data (databases, filesystems, secrets managers, etc) or tight dependencies (config services, databases, and other APIs come to mind).

example. i once worked with a team where we tried to get their app running locally in docker. (they used pcf back when it was called that; it's called tas now.) their app needed to use a dev instance of a db when it was not in a prod env. we asked if we could get a mocked schema. they said yes, but it would take three days.

it took three days because another team would manually produce the dataset from querying prod and modifying values. since they loaded it into the dev/staging environments, teams just used that. leadership also had no way of knowing whether devs were using data with real values on their workstations (because lack of automation and auditing), so politics were involved in producing a local schema that we could load into Postgres on Compose. (this was a financial company, so any environment with PII is fair game for auditors, which costs time and money.)

we landed up reverse-engineering the tables they needed so we could produce fake data good enough for integration to pass, but of course that introduces environment stratification of another kind since this team didn't own the data.

honestly, now that i wrote this, if every CTO forced their teams to make their core applications 12-factor, then staging environments would go away naturally while improving code quality and platform safety.

Re: We don’t use a staging environment

#295

Post by bcherny »

kgeist wrote: post #73
rock_hard wrote: post #8

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables…

How DB migrations work in particular (other things, like business logic, work similarly): https://news.ycombinator.com/item?id=29046303

Re: We don’t use a staging environment

#296

Post by hinkley »

rock_hard wrote: post #8

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

Problem I always seem to run into is that these optional features always seem to be added at a rate that's a bit higher than the rate at which the flags are retired. It doesn't take much of a multiplier for the number to become untenable pretty quickly.

Re: We don’t use a staging environment

#297

Post by hinkley »

toast0 wrote: post #170
kgeist wrote: post #73

Earlier quoted context omitted.

>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables…

> Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. I didn't work on FB databases while I was employed by them, but when you've got a lo…

For minor changes there's a simpler path, where you add a new field to the database, default the value to some reasonable value, then add it to the workflow in stages.

Depending on how your database feels about new columns and default values, there may be additional intermediate steps to keep it happy.

Re: We don’t use a staging environment

#298

Post by atmosx »

kgeist wrote: post #148
drewcoo wrote: post #85

Earlier quoted context omitted.

> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the…

>The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev enviro…

> because we routinely clone production data (deanonymized)

Are you using an external service or in-house tool to perform this operation?

Re: We don’t use a staging environment

#299

Post by reissbaker »

rock_hard wrote: post #8

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

It's common, but it's more like "this is common at young companies where the cost of maintaining staging can't pay for itself in improved productivity, because there aren't enough engineers for a 5% productivity improvement to be worth hiring multiple engineers for."

I'm sure that at FB at one point there wasn't a staging env. Today at FB there are multiple layers of staging, checked automatically as part of the deployment process. I'm ex-FB as well, and we definitely used staging environments every single day as part of the ordinary deploy pipeline. You probably worked there when it was younger, and smaller, and the tooling was less advanced.

Large tech companies have advanced dev tooling; eventually, the cost of paying people to make the tools is paid for in productivity gained per engineer, with large enough eng team sizes.

Re: We don’t use a staging environment

#300

Post by konaraddi »

This works for services that are growing slowly in features or have few other services integrating with it. I’m not sure how this scales when there are multiple services across multiple teams with dependencies on one another where services are being rapidly developed with new features. At work, we have staging/pre-prod environments across most teams that my team works with so new features can be tested in staging and other teams can test integrating with it. This is also possible to do with just a production environment but requires some engineering effort to add feature flags and special headers indicating a request is from a team looking to try a new API.
Post reply on HN