This probably also depends on your core business. If your product does not deal with real money, crypto, or other financial instruments and it is not serious if something goes wrong with a small number of people in production, this may work for you. It is probably cheaper and simpler.
Lots of products are not like that. I built a bank and work on stock exchanges. Probably not a good idea to save money by not testing as people get quite annoyed when their money goes missing.
We don’t use a staging environment
171–180 of 357 posts
Re: We don’t use a staging environment
#172Problem TL;DR:
"With staging:
- There could be differences from production
- Multiple people can't test at the same time
- Devs don't test their code."
Solution TL;DR: "Test your code, and push to production."
They completely misunderstood the problem and their solution literally changed nothing other than making devs test their code now. Staging could stay as is and would provide some significant risk mitigation with zero additional effort.
"Whenever we deploy changes, we monitor the situation continuously until we are certain there are no issues."
I'm sure customers would stay on the site, monitoring the situation too. Good luck with that strategy.
Re: We don’t use a staging environment
#173Problem TL;DR: "With staging: - There could be differences from production - Multiple people can't test at the same time - Devs don't test their code." Solution TL;DR: "Test your code, and push to production." They completely misunderstood the problem and their solution literally changed nothing other than making devs test their code now. Staging could stay as is and would provide some significant risk mitigation wit…
or they could maybe use a specific OS as their golden image, use ansible or chef or puppet or any of the hundreds of tools that config machines and keep their staging and prod in sync. Bonus points for introducing a service that produces mock data for staging.
Re: We don’t use a staging environment
#174This is good insofar as it forces you to make local development possible. In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally. That leads to people only testing in staging envs, causing staging to constantly break and discouraging automated tests that prevent regression bugs. It also leads to increasing complexity and interconn…
Ehh... once your systems use more than a few pieces of cloud infrastructure / SaaS / PaaS / external dependencies / etc, purely local development of the system is just not possible. There are some (limited) simulators / emulators / etc available and whatnot for some services, but running a full platform that has cloud dependencies on a local machine is often just not possible.
Forcing developers to deal with mocks right from the beginning is critical in my opinion. Unit testing as part of your CI/CD flow needs to be a first priority rather than something that gets thought of later on. Testing locally should be synonymous with running your unit test suite.
Doing your integration testing deployed to a non-production cloud environment is always necessary but should never be a requirement for doing development locally.
Re: We don’t use a staging environment
#175It really depends
Without staging environment, your chance of finding critical bugs rely on offline testing. Not all bugs can be found in unit tests, you need load tests to detect certain bugs that doesn't break your program from correctness perspective, but on latency/memory leakage front. And such tests might take longer time to run.
Staging slows things down, but it is intended, it creates a buffer to observe behavior. Depending on the nature of your service, it can be quite critical.
Re: We don’t use a staging environment
#176> We only merge code that is ready to go live. In their perception, is the rest of tech industry gambling in every pull request that some untested code would work in production? I work at a large company. We extensively test code on local machines. Then dev test environments. Then small roll out to just a few data centers in prod bed. Run small scale online flight experiments. Then roll out to the rest of prod bed. A…
Yeah, it seems like someone took RFC 9225 to heart. ( https://www.rfc-editor.org/rfc/rfc9225.html )
I wouldn't be surprised. I've seen colleagues reference April's Fools RFCs, and the reference wasn't meant to be taken as a joke.
Re: We don’t use a staging environment
#177Earlier quoted context omitted.
Having staging always encourages this. It’s really difficult to replicate prod in any non trivial way that exceeds what can be created on a workstation. Eg. Even if you buy the same hardware you can’t replicate production load anyway because it’s not being used by 5 million people concurrently. Your cache access patterns aren’t the same, etc. It’s far better to have a fast path to prod than a staging environment in m…
I think it's too much to expect staging to match the load and access patterns of your prod system. I find staging to be very useful. In various teams I have been a part of, I have seen the following productive use cases for staging 1. Extended development environment - If you use a micro-services or serverless architecture, it becomes really useful to do end-to-end tests of your code on staging. Docker helps locally,…
dev workstations should cost at least $4000.
Like how much productivity is being wasted because their machine is slow.
$4000 workstations are cheap compared to staging.
Re: We don’t use a staging environment
#178Imagine writing this entire blog post and being completely wrong about every topic you discuss. This is the most amateur content I've seen make it to the front page, let alone top post.
Re: We don’t use a staging environment
#179Earlier quoted context omitted.
It's pretty easy to create your own amis with developer versions. It makes sense why AWS doesn't necessarily provide this out of the box. But it still stands for fully managed versions of licensed software, you'll pay for the license even if it's non-prod
Yes, that's not to say it is not possible to create a similar env, but I thought the debate was how precisely you are replicating your production env. Sure it may be "good enough", but I thought the debate was about precision. How your own ami setup may differ from the AWS built from the developer version compared to the AWS ami? I don't know. Trying for an identical setup in staging is expensive, this is just a scen…
I was thinking about this line from the article:
> More often than not, each environment uses different hardware, configurations, and software versions.
They can't even deploy the same software versions to their staging environment. We're a long way off talking about precisely replicating load characteristics
Re: We don’t use a staging environment
#180Earlier quoted context omitted.
I am curios, why do you think it's impossible? I think we can establish that the database is the biggest culprit in making this difficult. As an independent developer, I have seen several teams that either back sync the prod db into the staging db OR capture known edge cases through diligent use of fixtures. I am not trying to counter your point necessarily, but just trying to understand your POV. Very possible that,…
The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors. There's a certain amount of "best practices"…
As if this wasn't enough of a headache, GDPR regulation requires more safeguards before you can put your prod-data in a secured staging environment.
Then there is the database size, which can make it hard and expensive to keep preprod up to date.
And should you want to measure performance, then no one else can use preprod while that is going on.