Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

181–190 of 357 posts

Re: We don’t use a staging environment

#181

Post by nickelpro »

lamontcg wrote: post #143
saq7 wrote: post #126

Earlier quoted context omitted.

I am curios, why do you think it's impossible? I think we can establish that the database is the biggest culprit in making this difficult. As an independent developer, I have seen several teams that either back sync the prod db into the staging db OR capture known edge cases through diligent use of fixtures. I am not trying to counter your point necessarily, but just trying to understand your POV. Very possible that,…

The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors. There's a certain amount of "best practices"…

The article doesn't talk about any of that though. The article says staging diffs prod because of:

> different hardware, configurations, and software versions

The hardware might be hard or expensive to get an exact match for in staging (but also, your stack shouldn't be hyper fragile to hardware changes). The latter two are totally solvable problems

Re: We don’t use a staging environment

#182

Post by karmasimida »

drewcoo wrote: post #90
repler wrote: post #86

Earlier quoted context omitted.

Exactly. “Staging never matches Prod” - well why is that? Make it so!!

I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.

It is possible.

But you need infrastructure and paying delicate attention to this problem. It is hard to define exactly what does replicating prod mean. And sometimes it might be difficult, e.g. prod might have access controlled customer data store that has its own problem, or it is about cost. But doesn't necessarily mean if you can't replicate perfectly, it is useless, you can still catch problems with things that you can replicate and do go wrong.

Ofc it is impossible to catch bugs 100% with staging, however, that argument goes either way.

Re: We don’t use a staging environment

#184

Post by pigcat »

blorenz wrote: post #131

We duplicate the production environment and sanitize all the data to be anonymous. We run our automated tests on this production-like data to smoke test. Our tests are driven by pytest and Playwright. God bless, I have to say how much I love Playwright. It just makes sense.

This is my first time hearing about Playwright. Curious to know what you like about it over other frameworks? I didn't glean a whole lot from the website.

Re: We don’t use a staging environment

#185

Post by sedatk »

klabb3 wrote: post #23

Not endorsing this point blank but.. One positive side effect of this is that it becomes much easier to rally folks into improving the fidelity of the dev environment, which has compound positive impact on productivity (and mental health of your engineers). In my experience at Big Tech Corp, dev environments were reduced to low unit test fidelity over years, then as a result you need to iterate (ie develop) in a stag…

They're not mutually exclusive. You can achieve local + staging environments at the same time. Stable local env + staging. Local is almost always the most comfortable option due to fast iteration times, so nobody would bother with staging by default. Make it good, people will come.

Re: We don’t use a staging environment

#186

Post by freedomben »

derefr wrote: post #121
laurent123456 wrote: post #114

Earlier quoted context omitted.

I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.

> Otherwise it wouldn't be possible to have migrations that work with both old and new code. Sure you can. Say that you've changed the type of a column in an incompatible way. You can, within a migration that executes as an SQL transaction: 1. rename the original table "out of the way" of the old code 2. add a new column of the new type 3. run an "INSERT ... SELECT ..." to populate the new column from a transformatio…

I wrote a blog about this for anyone who would like to learn more.

The query strings get you around the paywall if it comes up:

https://freedomben.medium.com/the-rules-of-clean-and-mostly-...

If anyone doesn't know what migrations are:

https://freedomben.medium.com/what-are-database-migrations-5...

Re: We don’t use a staging environment

#187

Post by darkwater »

lamontcg wrote: post #143
saq7 wrote: post #126

Earlier quoted context omitted.

I am curios, why do you think it's impossible? I think we can establish that the database is the biggest culprit in making this difficult. As an independent developer, I have seen several teams that either back sync the prod db into the staging db OR capture known edge cases through diligent use of fixtures. I am not trying to counter your point necessarily, but just trying to understand your POV. Very possible that,…

The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors. There's a certain amount of "best practices"…

IME, when you are not webscale, the issues you will miss from not testing in staging are bigger than the other way round. But that doesn't mean that all the extra efforts you have to put in the "test in prod only" scenario should not be put even when you do have a staging env.

Re: We don’t use a staging environment

#188

Post by sedatk »

kafrofrite wrote: post #173
sedatk wrote: post #172

Problem TL;DR: "With staging: - There could be differences from production - Multiple people can't test at the same time - Devs don't test their code." Solution TL;DR: "Test your code, and push to production." They completely misunderstood the problem and their solution literally changed nothing other than making devs test their code now. Staging could stay as is and would provide some significant risk mitigation wit…

or they could maybe use a specific OS as their golden image, use ansible or chef or puppet or any of the hundreds of tools that config machines and keep their staging and prod in sync. Bonus points for introducing a service that produces mock data for staging.

Yeah, and not achieving 100% parity is definitely not worth throwing away the benefits from, say, 80% parity.

Re: We don’t use a staging environment

#189

Post by toast0 »

saq7 wrote: post #132
_3u10 wrote: post #43

Earlier quoted context omitted.

Having staging always encourages this. It’s really difficult to replicate prod in any non trivial way that exceeds what can be created on a workstation. Eg. Even if you buy the same hardware you can’t replicate production load anyway because it’s not being used by 5 million people concurrently. Your cache access patterns aren’t the same, etc. It’s far better to have a fast path to prod than a staging environment in m…

I think it's too much to expect staging to match the load and access patterns of your prod system. I find staging to be very useful. In various teams I have been a part of, I have seen the following productive use cases for staging 1. Extended development environment - If you use a micro-services or serverless architecture, it becomes really useful to do end-to-end tests of your code on staging. Docker helps locally,…

> I think it's too much to expect staging to match the load and access patterns of your prod system.

For a lot of things, this makes staging useless, or worse. When production falls over, but it worked in staging, then staging gave unwarranted confidence. When you push to production without staging, you know there's danger.

That said, for changes that don't affect stability (which can sometimes be hard to tell), staging can be useful. And I don't disagree with a staging environment for your usecases.

Re: We don’t use a staging environment

#190

Post by freedomben »

_3u10 wrote: post #177
saq7 wrote: post #132

Earlier quoted context omitted.

I think it's too much to expect staging to match the load and access patterns of your prod system. I find staging to be very useful. In various teams I have been a part of, I have seen the following productive use cases for staging 1. Extended development environment - If you use a micro-services or serverless architecture, it becomes really useful to do end-to-end tests of your code on staging. Docker helps locally,…

dev workstations should cost at least $4000. Like how much productivity is being wasted because their machine is slow. $4000 workstations are cheap compared to staging.

when I worked for big corp, the reason we were told in engineering for getting $1,000 laptops was that it wasn't fair to accounting, HR, etc for us to have better machines. In the past people from these departments complained quite a bit.

The official reason (which was BS) was "to simplify IT's job by only having to support one model"

Post reply on HN