Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

261–270 of 357 posts

Re: We don’t use a staging environment

#261

Post by throwaway787544 »

This is good practice, except that blue/green is not exactly what you want. You want a smart load balancer that can shuffle an exact amount of traffic to a new service with your new deploy version. It must then evaluate the new service for errors and metrics, and then do an increase in shuffling traffic, etc, until you reach 100% shuffled traffic, at which time the old services can be decommissioned.

If at any time the monitoring of logs or metrics becomes unusual, it must shuffle all traffic away from the new service, alert devs, and halt all deploys (because someone needs to identify the bad code and unmerge it, thus requiring rework for all the subsequent work about to be merged). This is called "pulling the andon cord".

It is sad that there's all these comments saying this doesn't work. This has been the best practice established by Etsy, Martin Fowler, and others in the DevOps community for... 10 years? I guess until you see it for yourself it seems unbelievable. It requires a radical shift in design, development and operation, but it works great.

Re: We don’t use a staging environment

#262

Post by TruthWillHurt »

otterley wrote: post #37

The short answer appears to be "we are cheap and nobody cares yet." It's easy to damn the torpedoes and deploy straight into production if there's nobody to care about, or your paying customers (to the extent you have any) don't care either. Once you start gaining paying customers who really care about your service being reliable, your tune changes pretty quickly. If your customers rely on data fidelity, they're goin…

..And laptops *definitely* don't look like production.

Re: We don’t use a staging environment

#263

Post by ehnto »

rock_hard wrote: post #8

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

I think it's a difference in relationship with your users. For software I'm currently working on, we require verification that the changes we made were correct before they make it out to production, so there's a requirement for a staging environment they can access. That's a software > business relationship, where there's a known good outcome. This was also true in the ecommerce agency environments I worked in, the business owners want the opportunity to verify and correct things before they go out to production.

If it were a product > user relationship, where you're the product owner and you are trying to improve your product without an explicit request from your users, I can see how no staging environment makes sense. You have no responsibility of proof of correctness to your users, what you put out is what they get, and breakages can be handled as fixes after the fact.

Re: We don’t use a staging environment

#264

Post by throwaway787544 »

repler wrote: post #86
nickelpro wrote: post #24

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…

Exactly. “Staging never matches Prod” - well why is that? Make it so!!

For many, many cases, it is literally impossible, if not incredibly unrealistic. Doing away with it eliminates all of the problems associated with it and allows you to gain more confidence in your deploys. It simplifies while adding reliability and velocity. This is why abandoning staging is the best strategy.

Re: We don’t use a staging environment

#265

Post by ehnto »

drewcoo wrote: post #90
repler wrote: post #86

Earlier quoted context omitted.

Exactly. “Staging never matches Prod” - well why is that? Make it so!!

I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.

I've worked at a number of places where we had multiple products with replicated staging/production environments. One approach is to have your environments codified, prod gets built from the same pipeline staging does, and to automate database refreshes from prod > staging so they don't fall behind. It isn't rocket science, but of course it doesn't come at zero cost. Some production environments are pretty hard to replicate too, like anything with third party integrations that don't offer a staging environment of their own.

If you're a small shop I can understand it, but bigger companies with infrastructure teams, there's no excuse really, the technologies are all there.

Re: We don’t use a staging environment

#266

Post by mannykannot »

This appears to be just a naming convention issue. All the potential problems of staging environments can occur, for the same underlying reasons, in the approach advocated here, but they don't happen in staging merely because there isn't anything called that.

Personally, I think the approach advocated here is feasible, and even necessary if you are operating at global scale, but I am skeptical of tendentious stories about how it makes a number of problems just disappear.

Re: We don’t use a staging environment

#267

Post by rtpg »

kgeist wrote: post #208
toast0 wrote: post #170

Earlier quoted context omitted.

> Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. I didn't work on FB databases while I was employed by them, but when you've got a lo…

>Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. Our deployments aren't strictly "synchronous" either. We have thousands of database shards which are all migrated one by one (with some degree of parallelism), and new code is deployed only after all the shards ha…

Serious question: are you going to catch "corrupt data"-style migrations in staging in general?

There are of course "locks up the DB"-style migrations where you can then go in and fix it, so staging helps with that. But "oh this data is now wrong"-style errors seem to not really bubble up when you are just working off of test data.

Not to dismiss staging testing that much, but it feels like a tricky class of error where the answer is "be careful and don't delete data if you can avoid it"...

Re: We don’t use a staging environment

#268

Post by mtoddsmith »

bob1029 wrote: post #168
jasonhansel wrote: post #137

This is good insofar as it forces you to make local development possible. In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally. That leads to people only testing in staging envs, causing staging to constantly break and discouraging automated tests that prevent regression bugs. It also leads to increasing complexity and interconn…

> In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally At one time this was a huge blocker for our productivity. Access to a reliable test environment was only possible by way of a specific customer's production environment. The vendor does maintain a shared 3rd party integration test system, but its so far away from a realistic…

This is what we do as well. We just stub out the 3rd party integration and inject a dynamic configuration to generate whatever type of response we need.

Re: We don’t use a staging environment

#269

Post by __turbobrew__ »

devmunchies wrote: post #164

One approach I’m experimenting with is that all services communicate via a message channel (e.g. NATS or Pub/Sub). By doing this, I can run a service locally but connect it to the production pubsub server and then see how it effects the system if I publish events to it locally. I could also subscribe to events and see real production events hitting my local machine.

Yea that sounds like a nice way to do things. I could see there being security concerns that devs can directly access data streams from their local setup. For places with data controls I could see this being a no-go.

I guess you could have an anonymizer which consumes the production pub-sub and then anonymizes the data for consumption by non-prod environments.

Re: We don’t use a staging environment

#270

Post by mtoddsmith »

RandomBK wrote: post #259
WYepQ4dNnG wrote: post #17

I don't see how this can scale beyond a single service. Complex systems are made of several services and infrastructure all interconnected. Things that are impossible to run on local. And even if you can run on local, the setup is most likely very different from production. The fact that things work on local give a little to zero guarantees that they will work in prod. If you have a fully automated infrastructure set…

Note that this also necessarily requires that critical variables and configuration options are stored in version control, rather than database. Bootstrapping staging databases with values necessary to run the application is a constant challenge. Otherwise, your production environment would have massively different feature flags and other config than staging.

There are tools that can perform a diff of your databases and generate a change script. So we just diff local vs staging and capture the changes and check that in along with the code changes. Every change to the database create a schema change record so its easy to apply only the latest changes if they're versioned.
Post reply on HN