Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

141–150 of 357 posts

Re: We don’t use a staging environment

#141

Post by harunurhan »

sillysaurusx wrote: post #16
abhishekjha wrote: post #14

Earlier quoted context omitted.

That would be controlling a lot of feature flags given how many can be switched on at once. How do you control them?

flag = true More seriously, at my old company they just never got removed. So it wasn’t really about control. You just forgot about the ones that didn’t matter after awhile. If that sounds horrible, that’s probably the correct reaction. But it’s also common. Namespacing helps too. It’s easier to forget a bunch of flags when they all start with foofeature-.

> the ones that didn’t matter after awhile.

Ideally you have metrics for all flags and their values, so you can easily tell if one becomes redundant and safe to remove entirely after a while.

I've also seen making it a requirement to remove a flag after N days, the feature is completely rolled out.

Re: We don’t use a staging environment

#142

Post by jasonhansel »

Negitivefrags wrote: post #125

If you are saying you don't have a staging environment, what you are really saying is that your company doesn't have any QA process. If your QA process is just developers testing their own shit on their local machine then you are not going to get as much value out of staging.

But you don't need to have a single staging env shared by all QA testers. Why not create individual QA environments on an as-needed basis for testing specific features? Of course this requires you to invest in making it easy to create new environments, but it allows QA teams to test different things without interfering with each other.

Re: We don’t use a staging environment

#143

Post by lamontcg »

saq7 wrote: post #126
drewcoo wrote: post #90

Earlier quoted context omitted.

I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.

I am curios, why do you think it's impossible? I think we can establish that the database is the biggest culprit in making this difficult. As an independent developer, I have seen several teams that either back sync the prod db into the staging db OR capture known edge cases through diligent use of fixtures. I am not trying to counter your point necessarily, but just trying to understand your POV. Very possible that,…

The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors.

There's a certain amount of "best practices" effort you can go through in order to make your preprod environments sufficiently prod like but scaled down, with real data in their databases, running all the correct services, you can have a load testing environment where you hit one front end with a replay of real load taking from prod logs to look for perf regressions, etc. But ultimately time is better spent using feature flags and one box tests in prod rather than going down the rabbit hole of trying to simulate packet-level network failures in your preprod environment to try to make it look as prodlike as possible (although if you're writing your own distributed database you should probably be doing that kind of fault injection, but then you probably work somewhere FAANG scale, or you've made a potentially fatal NIH/DIY mistake).

Re: We don’t use a staging environment

#144

Post by tedmiston »

jasonhansel wrote: post #137

This is good insofar as it forces you to make local development possible. In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally. That leads to people only testing in staging envs, causing staging to constantly break and discouraging automated tests that prevent regression bugs. It also leads to increasing complexity and interconn…

Ehh... once your systems use more than a few pieces of cloud infrastructure / SaaS / PaaS / external dependencies / etc, purely local development of the system is just not possible.

There are some (limited) simulators / emulators / etc available and whatnot for some services, but running a full platform that has cloud dependencies on a local machine is often just not possible.

Re: We don’t use a staging environment

#145

Post by donohoe »

It seems like an April 1st troll (based on publication date), but I am assuming its not.

I can only say that this is a fairly poor decision from someone who appears knowledgeable to know better.

They could do everything they are doing as-is in terms of process, and just add a rudimentary test on a Staging environment as it passes to Production.

Over a long enough timeline it will catch enough critical issues to justify itself.

Re: We don’t use a staging environment

#146

Post by gabrieledarrigo »

ohmanjjj wrote: post #65

I’ve been shipping software for over two decades, built multiple successful SaaS companies, and have never in my life written a single unit test.

I feel no confident at all without unit tests on my code. Do you rely on some other types of testing?

Re: We don’t use a staging environment

#147

Post by raffraffraff »

user3939382 wrote: post #11

> Pre-live environments are never at parity with production Then you fix that particular problem. Infrastructure as code is one idea just off the top of my head.

Yup. If you have 4 production data centers, I imagine they're different sizes (autoscaling groups, Kubernetes deployment scale, perhaps even database instance sizes). So just build a staging environment that's like those, except smaller and not public. If you can't do that, then I'm willing to bet you can't deploy a new data center very quickly either, and your DR looks like ass.

Re: We don’t use a staging environment

#148

Post by kgeist »

drewcoo wrote: post #85
kgeist wrote: post #73

Earlier quoted context omitted.

>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables…

> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the…

>The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version

Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place.

As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev environment but in production it would lead to timeouts in requests for the duration of the migration for large clients (our application is sharded per tenant) because the table was very large for some of them and the migration locked it. The staging environment helped us find the problem before hitting production because we routinely clone production data (deanonymized) of the largest tenants to find problems like this. It's not practical (and maybe not very legal too) to force every developer have an up-to-date copy of that database on every VM/laptop, and load tests in an environment very similar to production show more meaningful results overall. And feature flags wouldn't help either because they only guard code. So far I'm unconvinced, it sounds pretty risky to me to go straight to prod.

I agree however that the concern about conflicts between feature toggles is largely a monolith problem, it's a communication problem when many teams make changes to the same codebase and are unaware of what the other teams are doing.

Re: We don’t use a staging environment

#149

Post by jokethrowaway »

A previous client was paying roughly 50% of their AWS budget (more than a million per year) just to keep up development and staging.

They were roughly 3x machines for live, 2x for staging and 1x for development.

Trying to get rid of it didn't work politically, because we had a cyclical contract with AWS where we were committing to spend X amount in exchange for discounts. Also, a healthy amount of ego and managers of managers BS.

In terms of what that company was doing, I'm pretty sure I could have exceeded their environment for 2k per month on hetzner (using auction).

Re: We don’t use a staging environment

#150

Post by aurbano »

Negitivefrags wrote: post #125

If you are saying you don't have a staging environment, what you are really saying is that your company doesn't have any QA process. If your QA process is just developers testing their own shit on their local machine then you are not going to get as much value out of staging.

I've seen this before at very large companies. All testing done in local and very little manual smoke testing in QA by either the PM or other engineers.

There are big tech companies that don't have QA people.

Post reply on HN