This is good insofar as it forces you to make local development possible. In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally. That leads to people only testing in staging envs, causing staging to constantly break and discouraging automated tests that prevent regression bugs. It also leads to increasing complexity and interconn…
Ehh... once your systems use more than a few pieces of cloud infrastructure / SaaS / PaaS / external dependencies / etc, purely local development of the system is just not possible. There are some (limited) simulators / emulators / etc available and whatnot for some services, but running a full platform that has cloud dependencies on a local machine is often just not possible.
We don’t use a staging environment
221–230 of 357 posts
Re: We don’t use a staging environment
#222This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states
Re: We don’t use a staging environment
#223Earlier quoted context omitted.
> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the…
>The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev enviro…
This doesn't have to be true. You can create an entirely separate table with the new data. New code knows how to join on this table, old code doesn't and thus ignores the new data. It doesn't work for every kind of migration, but in my experience, it's preferred by some DBAs if you have billions and billions of rows.
Example: `select user_id, coalesce(new_col2, old_col2) as maybe_new_data, new_col3 as new_data from old_table left join new_table using user_id limit 1`
Re: We don’t use a staging environment
#224This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states
Was this true for the systems that related to revenue and ad sales as well? While I can believe that a lot of code at Facebook goes into production without first going through a staging environment, I would be extremely surprised if the same were true for their ads systems or anything that dealt with payment flows.
Re: We don’t use a staging environment
#225This strategy won't scale beyond a very small team and codebase. The reasons mentioned, such as parity, are worth fixing.
Re: We don’t use a staging environment
#226> Pre-live environments are never at parity with production
My experience is that is is fairly trivial to have feature parity with production. Whatever you do for production, just do it again for staging. That's what it is meant to be.
> Most companies are not prepared to pay for a staging environment identical to production
Au contraire. All companies I've been to are more than willing to pay this. And secondly, it is pennies compared to production environment costs, because it isn't expected to handle any significant load. And, the article does mention being able to handle load as being one of the things that differ. I have not yet found the need to use changes to staging to verify load scaling capabilities.
> There’s always a queue
I don't undestand this paraph at all. It seems like an artificial problem created by how they handle repository changes, and has little to do with the purpose of a staging environment. It smells fishy to have local changes rely on a staging environment. The infrastructure I set up had a development environment be spun up and used for a development testing pipeline. Doesn't, and shouldn't need to rely on staging.
> Releases are too large
Well... one of the main benefits of having a staging environment is to safely do frequent small deployments. So this just seems like the exact wrong conclusion.
> Poor ownership of changes
This again, is not at all how I understand code should be shipped to a staging environment. "I’ve seen people merge, and then forget that their changes are on staging". What does this even mean? Surely, staging is only ever something that is deployed to from the latest release branch, which also surely comes from a main/master? The following "and now there are multiple sets of changes waiting to be released", also suggest some fundamental misunderstanding. *Releases* are what are meant to end up in staging. should be *a* release.
> People mistakenly let process replace accountability > "By utilising a pre-production environment, you’re creating a situation where developers often merge code and “throw it over the fence"
Again. Staging environment isn't a place where you dump your shit. "Staging" is a place where releases are verified in an as much-as-possible-the-same-environment-as-production. So, again. This seems like entirely missing the point.
----
It seems to me that they don't use a staging environment, because they don't understand what such a thing should be used for. I'd be completely OK with someone rationalizing this as "too much of a hassle". But to try and justify something so poorly...
From their conclusion:
> Dropping your staging environment in favour of true continuous integration and deployment can create a different mindset for shipping software. When there is no buffer for changes before they go live, you need to be confident that your changes are fit for production. You also need to be alert and take full ownership of any changes you make.
Well... of course there is a shift in mindset when you'll be shitting your pants every time you make a change in production, since that's when you'll get to see if you broke something. The whole point of a staging environment is to have a buffer.... so that you don't have to be "confident". So that you don't have to be on high alert, because you have alerts that can trigger without anything important going offline. So that ownership isn't crucial in a post-fuck-up blame game.
Re: We don’t use a staging environment
#227Re: We don’t use a staging environment
#228This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states
Facebook can completely break the user experience for 4.3 million different users each day and each user would only experience one breakage per year. This is pretty common, but not because most employing it have 1.6bn users and 10k engineers; essentially enough scale to throw bodies at problems.
Re: We don’t use a staging environment
#229Earlier quoted context omitted.
> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the…
I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.
Re: We don’t use a staging environment
#230Earlier quoted context omitted.
Ehh... once your systems use more than a few pieces of cloud infrastructure / SaaS / PaaS / external dependencies / etc, purely local development of the system is just not possible. There are some (limited) simulators / emulators / etc available and whatnot for some services, but running a full platform that has cloud dependencies on a local machine is often just not possible.
The answer (IMHO) is to not use services that make it impossible to develop locally, unless you can trivially mock them; the benefits of such services aren't worth it if they result in a system that is inherently untestable with an environment that's inherently unreproducible. (I can go on a rant about AWS Lambda, and how if they'd used a standardized interface like FastCGI it would make local testing trivial, but th…