Live data from Hacker News

We don’t use a staging environment

squeaky.ai

341–350 of 357 posts

Re: We don’t use a staging environment

#341

I don't see how this can scale beyond a single service. Complex systems are made of several services and infrastructure all interconnected. Things that are impossible to run on local. And even if you can run on local, the setup is most likely very different from production. The fact that things work on local give a little to zero guarantees that they will work in prod. If you have a fully automated infrastructure set…

The problem with staging environments is that replicating the functionality is easy but replicating the data, interactions, and behavior of people in a real environment is not. It's better to think in terms of early access releases and some kind of controlled roll out of new software so you catch bugs and issues before they impact most of your users. I've seen many projects where the staging environment is a bad joke…

you need both, in my experience working in SaaS, enterprises expect reliable and stable platforms. A staging environment is that extra safety net that can help preventing shipping a completely broken product. In a staging env you can turn on/off experiments and feature flags before doing that in production.

That said, you should also build the product so that you can run experiment and only turn on a feature for a small sub set of production customers, usually the free tier. To then gradually rollout to everybody else.

Last, staging env should be considered a production-grade env, thus if it breaks there should be SRE/DEV on call ready to jump and fix it.

Re: We don’t use a staging environment

#342
post #216

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

The book Software Engineering at Google or something akin to that mentions the same kind of thing.

This one: "Site Reliability Engineering: How Google Runs Production Systems"?

Re: We don’t use a staging environment

#343

Earlier quoted context omitted.

People hold up banking as the pinnacle of serious and responsible high-quality high-reliability software engineering and operations, but my bank is the only web property or mobile app I use that's routinely unavailable for hours at a time.

Do you want your money managed by a systems that starts on local and goes straight to production?

Compared to a system with maintenance windows, yeah, maybe? Banking seems pretty forgiving honestly. Customers accept 3-5 business days latency per transaction. And even if you get it wrong the first time, money is fungible. It seems like it should call for more relaxed practices than systems that need to process non-fungible data in tens of milliseconds 24x7.

Re: We don’t use a staging environment

#344
post #216

Earlier quoted context omitted.

The book Software Engineering at Google or something akin to that mentions the same kind of thing.

This one: "Site Reliability Engineering: How Google Runs Production Systems"?

https://www.oreilly.com/library/view/software-engineering-at...

Re: We don’t use a staging environment

#345
post #95

Earlier quoted context omitted.

And yes, you need blue/green deployments in addition to feature flags, as it is not easy to feature flag certain things, such as a language runtime version update or a third party library upgrade, among many other things.

If you're running in containers aren't those version upgrades isolated?

The trick is to be able to route users traffic to different deployments. You can run two versions of your application concurrently, and have a dial to progressively shift traffic to the new version, as soon as you notice anything wrong you shift it back to the previous version which wasn't stopped at all.

After 100% of the traffic is in the new version, and no customer complaints for 1h then you can shut down the old version.

Google App Engine had all of this at least 5 or 6 years ago.

Re: We don’t use a staging environment

#346

Earlier quoted context omitted.

The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors. There's a certain amount of "best practices"…

The article doesn't talk about any of that though. The article says staging diffs prod because of: > different hardware, configurations, and software versions The hardware might be hard or expensive to get an exact match for in staging (but also, your stack shouldn't be hyper fragile to hardware changes). The latter two are totally solvable problems

I was responding to other commentors not really the title article.

The stuff you cite there is pretty simple to deal with, configuration management is basically a solved problem and IDK how you can't just fix the different hardware.

The more universal problem of making preprod look just like prod so that you have 100% confidence in a rollout without any of the testing-in-prod patterns (feature flags, smoke tests, odd/even rollouts, etc) is not very solvable though.

Re: We don’t use a staging environment

#348

Earlier quoted context omitted.

FWIW I don't think it is weird at all. Maybe a little short on details of what ready really means for example. While I don't think going completely staging-less makes a lot of sense, going without a shared staging environment is a good thing. It is absolutely awesome to be able to have your own "staging" environment for testing that is independent of everyone else. With the Cloud this is absolutely possible. Shared s…

> "staging" environment for testing that is independent of everyone else That's not usually what people mean by staging. Staging is a type of pre-production test environment where several different features that are continously developed by different teams can be tested together. Third party integrations such as logistics, ordering and payment systems can also have their integration testing here. > Shared staging env…

    Staging is a type of pre-production test environment where several different features that are continously developed by different teams can be tested together
Yes, that is exactly what we can do on these environments that any dev or QA has at their fingertips. We have multiple teams that work on several different features/parts of the application that sometimes live in the same services as other teams work on as well or they are smaller, more dedicated services and they're the only team working on those specifically but they still usually work together with other services in the overall system. Some of those have third party integrations. FWIW, in our case usually we are the ones testing integrations with a third party and not the other way around.

    That it can be a logistical challenge to test code that touches many parts of a multi stakeholder system does not mean we shouldn't do it.
As mentioned earlier, these individually deployable environments are fully functional and fully integrated. Now I understand that in some situations it can be hard to do a full integration with a third party, because the third party is not able to able to accommodate the numerous environments that you are able to provide. In those cases compromises might need to be made. That is overall a bad thing though. E.g. to take your "third party ordering system" example again. It would be best, if you could have a separate account/tenant/instance (whatever makes sense in the exact circumstances and nature of system) in said third party system but sometimes that might not be possible and a third party system might need to be shared somehow between all your own staging/dev envinronments.

    That sounds dangerously close to "testing is hard, let's go shopping". Having to wait weeks to test a feature sounds like the process has broken down, not that the process is unnecessary. 
That is never what I said. I said that we can create such staging environments that are a fully integrated set of services on our end really easily and that that is awesome to have. The "having to wait weeks on end" is something I have experienced at previous clients/employers and I absolutely agree with you that it's a broken system. I all too well remember the "yes, we can have INT-3 for 2 hours next Tuesday, do you think we can get all our tests done in that time frame? After that they need it for extensive performance testing for a week and INT-1 won't be available until Thursday at 11". And then you gotta answer "Sorry 2 hours is barely enough to do the deploy and re-configuring of the environment because we need to manually restore that special data set for the third party logistics system mock and then adjust the configuration and that alone takes those 2 hours if the issues we had last time are any indication".

Sooo much better to click the deploy button (or in the dev case we usually use the command line ;) ) and 10 minutes later you have your code and everything else deployed.

Re: We don’t use a staging environment

#349
post #267
post #208

Earlier quoted context omitted.

>Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. Our deployments aren't strictly "synchronous" either. We have thousands of database shards which are all migrated one by one (with some degree of parallelism), and new code is deployed only after all the shards ha…

Serious question: are you going to catch "corrupt data"-style migrations in staging in general? There are of course "locks up the DB"-style migrations where you can then go in and fix it, so staging helps with that. But "oh this data is now wrong"-style errors seem to not really bubble up when you are just working off of test data. Not to dismiss staging testing that much, but it feels like a tricky class of error wh…

Even the "locks up the DB" migration behavior tends to depend on the load & size of tables, which staging might not recreate.

Re: We don’t use a staging environment

#350

Earlier quoted context omitted.

FWIW I don't think it is weird at all. Maybe a little short on details of what ready really means for example. While I don't think going completely staging-less makes a lot of sense, going without a shared staging environment is a good thing. It is absolutely awesome to be able to have your own "staging" environment for testing that is independent of everyone else. With the Cloud this is absolutely possible. Shared s…

How do you get realistic data into these environments? Our production has 10TB of database data and 20M files uploaded by customers. We regularly see devs write queries or nested loops that work fine on their tiny dev system but explode when run against realistic customer data. Right now we go through a 12 hour restore then anonymization pass to keep staging up to date. Very painful. Who has a better way?

YMMV as always, i.e. this might be faster/easier to implement for some use cases/companies than others.

I would argue that for most changes in most companies it does not matter if you have a full data set equivalent of Prod. Especially since it's an ever growing target (hopefully for you :)). As we can see from your use case, that can create challenges. If we were testing every little change with a complete replica of our Prod environment we'd spend a lot more money on this and would also have to wait way too long for these environments to come up. This might then drive us towards keeping a small set of staging environments running all the time and share them.

Now I don't know your specific field, what kind of system you provide, what the cost of failures in Prod would be, what kind of guards you have in your systems against a complete outage like that (e.g. can a bad query running for one customer take out your entire product or will it be an isolated incident to that one customer or a small group of customers?) etc. If so maybe you first want to isolate failures between customers more.

FWIW, I also see devs write these kinds of queries and code and we catch most of them in PRs and most people learn from this and next time they don't write this kind of query any longer. But let's say they do and we don't catch them. Can you analyze your data set and figure out a small enough example that is not simply an ever growing anonymized version of actual Prod data that would exhibit the same catastrophic characteristics for most queries these devs might write? I'm pretty sure the kinds of queries that fail on 10TB of data w/ 20M files fail just as spectacularly on 9TB and 19M files? Have you tried if they also fail 'badly enough' on 500GB of data and 1M files? How fast would it be to restore a fixed data set like that? Maybe refresh it every month from Prod if that's needed for some reason or another.

A lot of what you might actually want to try out in the end will depend on your exact system set up I would argue and it's hard to give general advice that will definitely fit.

Post reply on HN