No disrespect but you can do this for an analytics dashboard or a content web site with canaries (Facebook). May not be the best for high liability sites like financial systems.
We don’t use a staging environment
241–250 of 357 posts
Re: We don’t use a staging environment
#242Earlier quoted context omitted.
We don't have a staging environment (for the backend) at work either. However, depending on the size of the tables in-question, a migration might take days. Thus, we usually ask DBA's for a migration days/weeks before any code goes live. There's usually quite a bit of discussion, and sometimes suggestions for an entirely different table with a join and/or application-only (in code, multiple query) join.
Sorry for the silly question, perhaps, but what is the purpose of a db migration? Do schemas in production change that often? For context, the last couple of services I wrote all have fixed, but implicit schema, (built on key value stores). That is, the DB has no types. So instead, the type system is enforced by the API layer. Any field changes so far are gated via API access and APIs have backwards compatibility con…
Relational databases can be very strict, for example if you use foreign key references then the data base enforces that a row exists in the referenced table for every foreign key in the referring table. This strict enforcement makes it difficult to change schema.
The way you handle things with API level enforcement is actually a good architecture and it would probably make schema changes easier to deal with even on a relational database backend.
Re: We don’t use a staging environment
#243No disrespect but you can do this for an analytics dashboard or a content web site with canaries (Facebook). May not be the best for high liability sites like financial systems.
People hold up banking as the pinnacle of serious and responsible high-quality high-reliability software engineering and operations, but my bank is the only web property or mobile app I use that's routinely unavailable for hours at a time.
Re: We don’t use a staging environment
#244This can pass for a startup but not for a proper business with customers.
Do you get audited?
Re: We don’t use a staging environment
#245This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states
Was this true for the systems that related to revenue and ad sales as well? While I can believe that a lot of code at Facebook goes into production without first going through a staging environment, I would be extremely surprised if the same were true for their ads systems or anything that dealt with payment flows.
I worked on ad systems at Facebook, and yes it's (approximately) true for those as well.
The thing to realize that "in production" almost never means "rolled out to 100% of users from 0%". Instead you'd do very slow rollouts to, say, 1% of people in Hungary (or whatever) and use a ton of automatic measurements over time as well as lots and lots of tests to validate that things were working as expected before rolling out a little more, then a little more after that. By the time the code is actually being hit by the majority of users, it's often been run billions of times already.
Re: We don’t use a staging environment
#246This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states
That would be controlling a lot of feature flags given how many can be switched on at once. How do you control them?
You have automated tools that yell at you to clean up feature flags and you force people to include sensible expiration dates at part of your PR process. Flags past the date result in increased yelling. If your team has too much crap in the codebase eventually someone politely tells you to clean it up.
You also have tooling that measures how many times a flag was encountered vs. how many times it actually triggered etc. Once it looks like it's at 100% of traffic, again you have automations that tell people to clean up their crap.
Re: We don’t use a staging environment
#247This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…
Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...
That's per server, not per database. I have dozens of staging databases on my sql server - product_name_pr_id.
Re: We don’t use a staging environment
#248Earlier quoted context omitted.
There's a difference between permanent staging environments that need maintenance and disposable "staging" environments that are literally a clone of what's on your laptop that you trash once UAT/smoke is done. The former costs money and can lie to you; the latter is literally prod, but smaller.
This makes it sound so easy, but in my experience, permanent staging environments exist because setting up disposable staging environments is too complex. How do you deal with setting up complex infrastructure for your disposable staging environment when your system is more complex than a monolithic backend, some frontend and a (small) database? If your system consists of multiple components with complex interactions…
If at all possible your entire infrastructure should be defined as code.
At my workplace we use aws cdk for infrastructure and standing up a new environment is as easy as calling ‘cdk deploy’ and then we have a script which runs after the provision to copy in data.
Re: We don’t use a staging environment
#249Staging is bloat. You want to get to production asap. Always deploy to production only.
Re: We don’t use a staging environment
#250Earlier quoted context omitted.
I think it's too much to expect staging to match the load and access patterns of your prod system. I find staging to be very useful. In various teams I have been a part of, I have seen the following productive use cases for staging 1. Extended development environment - If you use a micro-services or serverless architecture, it becomes really useful to do end-to-end tests of your code on staging. Docker helps locally,…
dev workstations should cost at least $4000. Like how much productivity is being wasted because their machine is slow. $4000 workstations are cheap compared to staging.
Multiply that by the number of employees who need it ... staging is cheaper.