Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

251–260 of 357 posts

Re: We don’t use a staging environment

#251

Post by throwaway6532 »

repler wrote: post #86
nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Exactly. “Staging never matches Prod” - well why is that? Make it so!!

If that's how it is at every single company then saying "just make it the same" probably isn't the answer.

Re: We don’t use a staging environment

#252

Post by strken »

laurent123456 wrote: post #114
cmeacham98 wrote: post #99

Earlier quoted context omitted.

I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.

I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.

My impression is that once you're at Facebook scale, most of your migrations are massive undertakings that need to take into account things like "How many terabytes more space do we need?", "How do we control the load on our DB nodes while the migration is going on?", "How do we get data from cluster A to cluster B?", "Is adding this index going to take hours and break everything?", and so on. Some of the time you'll be spinning up an entirely new cluster rather than changing the schema of an old one, and when you do migrate an existing cluster there's some five-page document specifying a week-long plan with different phases.

Then internally, they work around the inflexible db schemas by using offline batch processing tools or generic systems that can handle arbitrary data, for tasks that would be handled by the one DB in smaller systems.

Re: We don’t use a staging environment

#253

Post by sumedh »

toast0 wrote: post #189
saq7 wrote: post #132

Earlier quoted context omitted.

I think it's too much to expect staging to match the load and access patterns of your prod system. I find staging to be very useful. In various teams I have been a part of, I have seen the following productive use cases for staging 1. Extended development environment - If you use a micro-services or serverless architecture, it becomes really useful to do end-to-end tests of your code on staging. Docker helps locally,…

> I think it's too much to expect staging to match the load and access patterns of your prod system. For a lot of things, this makes staging useless, or worse. When production falls over, but it worked in staging, then staging gave unwarranted confidence. When you push to production without staging, you know there's danger. That said, for changes that don't affect stability (which can sometimes be hard to tell), stag…

> For a lot of things, this makes staging useless, or worse.

That depends on what Staging is used for, if its used to run e2e tests, giving a demo to PMs etc, you can use Staging. For performance testing you can setup a similar env like Prod, run your perf tests and then kill the perf env or you can scale up the staging env, dont let anyone use it except for performance and then scale it down.

Re: We don’t use a staging environment

#254

Post by deeteecee »

nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

> They're not arguing that testing or staging environments are bad, they're just saying their organization couldn't manage to get them working.

That is exactly what I got from reading this article. Their staging process was poorly set up and they simply abandoned ship. Additionally, I was getting poor software culture vibes.

Re: We don’t use a staging environment

#255

Post by WatchDog »

WYepQ4dNnG wrote: post #17

I don't see how this can scale beyond a single service. Complex systems are made of several services and infrastructure all interconnected. Things that are impossible to run on local. And even if you can run on local, the setup is most likely very different from production. The fact that things work on local give a little to zero guarantees that they will work in prod. If you have a fully automated infrastructure set…

I think the problems they have with managing non prod environments is actually a symptom of having many systems. Staging environments are easy to maintain when it’s one system, when you have a complicated service oriented architecture, it becomes much more difficult and expensive to maintain non prod environments.

Re: We don’t use a staging environment

#256

Post by ransom1538 »

derefr wrote: post #121
laurent123456 wrote: post #114

Earlier quoted context omitted.

I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.

> Otherwise it wouldn't be possible to have migrations that work with both old and new code. Sure you can. Say that you've changed the type of a column in an incompatible way. You can, within a migration that executes as an SQL transaction: 1. rename the original table "out of the way" of the old code 2. add a new column of the new type 3. run an "INSERT ... SELECT ..." to populate the new column from a transformatio…

Not to be rude but this isn't how this works at all. Things like 'run an "INSERT ... SELECT ..."' can't happen at scale due to locking. How they actually do it is super rad:

https://www.percona.com/doc/percona-toolkit/3.0/pt-online-sc...

tl;dr; They setup a system of triggers (updates,inserts,etc) , copy the data over, then run through all the data in the trigger system. percona developed all these fancy features as well to monitor replica data etc. Another way with cloud vms (terabyte+ tables), you image a replica, do the alter, let the replica catch up, image it, build replicas off that, promote this to master.

Facebook's internal one I hear is close to this.

Re: We don’t use a staging environment

#257

Post by _3u10 »

chupasaurus wrote: post #250
_3u10 wrote: post #177

Earlier quoted context omitted.

dev workstations should cost at least $4000. Like how much productivity is being wasted because their machine is slow. $4000 workstations are cheap compared to staging.

Multiply that by the number of employees who need it ... staging is cheaper.

Multiply that by the lost hours waiting for compilation, etc and whether you have staging or not doesn’t matter.

Re: We don’t use a staging environment

#258

Post by TruthWillHurt »

jasonhansel wrote: post #203
tedmiston wrote: post #144

Earlier quoted context omitted.

Ehh... once your systems use more than a few pieces of cloud infrastructure / SaaS / PaaS / external dependencies / etc, purely local development of the system is just not possible. There are some (limited) simulators / emulators / etc available and whatnot for some services, but running a full platform that has cloud dependencies on a local machine is often just not possible.

The answer (IMHO) is to not use services that make it impossible to develop locally, unless you can trivially mock them; the benefits of such services aren't worth it if they result in a system that is inherently untestable with an environment that's inherently unreproducible. (I can go on a rant about AWS Lambda, and how if they'd used a standardized interface like FastCGI it would make local testing trivial, but th…

Awesome. you just cost your company $500K in salaries for people to maintain databases, networks, storage, servers and a bunch of other stuff Google/AWS already do much better than you.

How lucky you are that management pays you to pursue your hobbies!

Re: We don’t use a staging environment

#259

Post by RandomBK »

WYepQ4dNnG wrote: post #17

I don't see how this can scale beyond a single service. Complex systems are made of several services and infrastructure all interconnected. Things that are impossible to run on local. And even if you can run on local, the setup is most likely very different from production. The fact that things work on local give a little to zero guarantees that they will work in prod. If you have a fully automated infrastructure set…

Note that this also necessarily requires that critical variables and configuration options are stored in version control, rather than database. Bootstrapping staging databases with values necessary to run the application is a constant challenge.

Otherwise, your production environment would have massively different feature flags and other config than staging.

Re: We don’t use a staging environment

#260

Post by ehnto »

Gigachad wrote: post #199
rock_hard wrote: post #8

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

This is how my current place does it. The only issue we are having is library / dependency updates have a tendency to work perfectly fine locally and then fail in production due to either some minor difference in environment or scale. It's a problem to the point that we have 5 year old ruby gems which have no listed breaking changes because no one is brave enough to bump them. I had a go at it and caused a major prod…

Do you have a replicate of the production environment codified somehow, like into a VM? It's rarely perfect, but I usually try and develop locally on the same stack I deploy to which can help with the environment differences.

It's also why I think it's smart to rebuild the environment on deploy if it makes sense for your pipeline, so that you wipe any minor differences that have been accruing over time. Working on a long running product, you quickly find yourself with disparities building up, and they're not codified, so they're essentially unknown to the team until they cause an issue.

Post reply on HN