Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

121–130 of 357 posts

Re: We don’t use a staging environment

#121

Post by derefr »

laurent123456 wrote: post #114
cmeacham98 wrote: post #99

Earlier quoted context omitted.

I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.

I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.

> Otherwise it wouldn't be possible to have migrations that work with both old and new code.

Sure you can. Say that you've changed the type of a column in an incompatible way. You can, within a migration that executes as an SQL transaction:

1. rename the original table "out of the way" of the old code

2. add a new column of the new type

3. run an "INSERT ... SELECT ..." to populate the new column from a transformation of existing data

4. drop the old column of the old type

5. rename the new column to the old column's name

6. define a view with the name of the original table, that just queries through to the new (original + renamed + modified) table for most of the original columns, but which continues to serve the no-longer-existing column with its previous value, by computing its old-type value from its new-type value (+ data in other columns, if necessary.)

Then either make sure that the new code is reading directly from the new table; or create a trivial passthrough view for the new version to use as well.

(IMHO, as long as you've got writable-view support, every application-visible "table" should really just be a view, with its name suffixed with the ABI-major-compatibility-version of the application using it. Then the infrastructure team — and more specifically, a DBA, if you've got one — can do whatever they like with the underlying tables: refactoring them, partitioning them, moving them to other shards and forwarding them, etc. As long as all the views still work, and still produce the same query results, it doesn't matter what's underneath them.)

Re: We don’t use a staging environment

#122

Post by shoo »

different business or organisational contexts have different deployment patterns and different negative impacts of failure.

in some contexts, failures can be isolated to small numbers of users, the negative impacts of failures are low, and rollback is quick and easy. in this kind of environment, provided you have good observability & deployment, it might be more reasonable to eliminate staging and focus more on being able to run experiments safely and efficiently in production.

in other contexts, the negative impacts of failure are very high. e.g. medical devices, mars landers, software governing large single systems (markets, industrial machinery). in these situations you might prefer to put more emphasis on QA before production.

Re: We don’t use a staging environment

#123

Post by adamredwoods »

drewcoo wrote: post #98
zimbatm wrote: post #70

If you can, provide on-demand environments for PRs. It's mostly helpful to test frontend changes, but also database migrations and just demoing changes to colleagues. If you have that, you will see people's behaviour change. We have a CTO that creates "demo" PRs with features they want to show to customers. All all the contension around staging as identified in the article is mostly gone.

You point out another kind of use of staging I've seen. "Don't touch staging until tomorrow after because SoAndSo is giving a demo to What'sTheirFace" so a bunch of engineering activity gets backed up.

We use multiple staging lambdas specifically for demos and QA. CICD with terraform. Works great.

Re: We don’t use a staging environment

#124

Post by dexwiz »

jurschreuder wrote: post #117
NorwegianDude wrote: post #108

Staging, tests, previews and even running code locally is for people who make mistakes. It's dumb and a total waste of time if you don't make any mistakes. No testing at all, that's what I call optimizing for success! On a more serious note: Sometimes staging is the same as local, and in those situations there is very limited use for staging.

We often deploy to production directly because a customer wants a feature right now. I was thinking of changing the staging server to be called beta. Customers can use new features directly, but at their own risk.

Staging environments should be separate from production environments. If the Beta is expected to persist data in the long term, then it's not staging. Staging environments should be nukable. You don't want a messy Beta release to corrupt production data or to have customers trying to sue you if you reset staging.

I don't know about your customer but wanting a feature yesterday may be a sign of some dysfunctional operating practices. Shortening your already short deployment pipeline shouldn't be your answer, unless its currently part of the problem. Otherwise, this should be solved with setting better expectations.

Re: We don’t use a staging environment

#125

Post by Negitivefrags »

If you are saying you don't have a staging environment, what you are really saying is that your company doesn't have any QA process.

If your QA process is just developers testing their own shit on their local machine then you are not going to get as much value out of staging.

Re: We don’t use a staging environment

#126

Post by saq7 »

drewcoo wrote: post #90
repler wrote: post #86

Earlier quoted context omitted.

Exactly. “Staging never matches Prod” - well why is that? Make it so!!

I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.

I am curios, why do you think it's impossible?

I think we can establish that the database is the biggest culprit in making this difficult.

As an independent developer, I have seen several teams that either back sync the prod db into the staging db OR capture known edge cases through diligent use of fixtures.

I am not trying to counter your point necessarily, but just trying to understand your POV. Very possible that, in my limited experience, I haven't come across all the problems around this domain.

Re: We don’t use a staging environment

#127

Post by shoo »

jurschreuder wrote: post #117
NorwegianDude wrote: post #108

Staging, tests, previews and even running code locally is for people who make mistakes. It's dumb and a total waste of time if you don't make any mistakes. No testing at all, that's what I call optimizing for success! On a more serious note: Sometimes staging is the same as local, and in those situations there is very limited use for staging.

We often deploy to production directly because a customer wants a feature right now. I was thinking of changing the staging server to be called beta. Customers can use new features directly, but at their own risk.

it's a good idea to be crystal clear about which environments are running production workloads. if you end up with "non-production" environments running production workloads then it becomes much easier to accidentally blow away customer data, let alone communicate coherently. "beta" is fine provided it is regarded as a production environment. you may still want a non-production staging environment!

i worked somewhere that had fallen into this kind of mess, where 80% of the business' production workloads were done in the Production environment, and 20% of the business' production workloads (with slightly different requirements) were done in a non-production test environment. it took calendar years to dig out of that hole.

Re: We don’t use a staging environment

#129

Post by jurschreuder »

dexwiz wrote: post #124
jurschreuder wrote: post #117

Earlier quoted context omitted.

We often deploy to production directly because a customer wants a feature right now. I was thinking of changing the staging server to be called beta. Customers can use new features directly, but at their own risk.

Staging environments should be separate from production environments. If the Beta is expected to persist data in the long term, then it's not staging. Staging environments should be nukable. You don't want a messy Beta release to corrupt production data or to have customers trying to sue you if you reset staging. I don't know about your customer but wanting a feature yesterday may be a sign of some dysfunctional oper…

It's mostly front-end features that change a lot, so there is not much danger in running them on the prod api and db. Our api is very stable because it uses event streaming. Mostly the front-end is different for different customers.

Re: We don’t use a staging environment

#130

Post by ninth_ant »

laurent123456 wrote: post #114
cmeacham98 wrote: post #99

Earlier quoted context omitted.

I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.

I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.

That is largely the case.

For other, more complex cases where that is not possible, you migrate a portion of the userbase to a new db schema and codepath at the same time.

Post reply on HN