Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

351–357 of 357 posts

Re: We don’t use a staging environment

#351

Post by tatersolid »

tharkun__ wrote: post #350
tatersolid wrote: post #337

Earlier quoted context omitted.

How do you get realistic data into these environments? Our production has 10TB of database data and 20M files uploaded by customers. We regularly see devs write queries or nested loops that work fine on their tiny dev system but explode when run against realistic customer data. Right now we go through a 12 hour restore then anonymization pass to keep staging up to date. Very painful. Who has a better way?

YMMV as always, i.e. this might be faster/easier to implement for some use cases/companies than others. I would argue that for most changes in most companies it does not matter if you have a full data set equivalent of Prod. Especially since it's an ever growing target (hopefully for you :)). As we can see from your use case, that can create challenges. If we were testing every little change with a complete replica o…

We have a cut-down but “production shaped” dataset for fast environment creation but even with code reviews it hasn’t been successful at catching the O(n^2) unindexed query or accidental O(scary) at the app tier caused by a refactor. Or the data migration which wasn’t batched but needs to be. Developer turnover is only accelerating, with predictable results.

At this point we are looking into volume snapshots as a potential speedup for creation of a full-sized per-release staging environments but that still leaves the problem of generating a realistic multi-tenant customer load to solve and maintain over time.

Re: We don’t use a staging environment

#352

Post by clintonb »

simonw wrote: post #283
clintonb wrote: post #81

Earlier quoted context omitted.

How do you define a large scale database migration? If you're just updating data or schema, that can be done locally via integration test. No need for a separate environment.

You have a table with 200m rows in it. You want to me extract one of the columns in that table out to a new, separate table - a job that will take several hours to complete. You want to do this without any visible downtime or breakage to your end-users - likely with some kind of complex dual-write and/or dual/read mechanism during that operation.

You make the change in multiple phases.

1. Add the new table. 2. Update code to write/read from both the new and old data sources (e.g., read from the new table first before falling back to the old column). 3. Copy data from the column to the new table. 4. Update code to stop using the old column. 5. Remove the old column from the database.

I wrote a blog post about this: https://medium.com/edx-engineering/django-migration-donts-f4....

Re: We don’t use a staging environment

#353

Post by rurban »

In one of my very first jobs in the mid 90ies there was also an incoming team we took over from a major competitor, who made the processes I introduced much simpler by removing dev testing, staging and CVS. They preferred to work as root on the live servers with 80.000 customers. Development was apparently so much easier with immediate feedback.

I liked that so much, that I resigned, and found a much better job 2 years later. I guess you could say bad cultural fit.

Re: We don’t use a staging environment

#354

Post by nprateem »

tatersolid wrote: post #351
tharkun__ wrote: post #350

Earlier quoted context omitted.

YMMV as always, i.e. this might be faster/easier to implement for some use cases/companies than others. I would argue that for most changes in most companies it does not matter if you have a full data set equivalent of Prod. Especially since it's an ever growing target (hopefully for you :)). As we can see from your use case, that can create challenges. If we were testing every little change with a complete replica o…

We have a cut-down but “production shaped” dataset for fast environment creation but even with code reviews it hasn’t been successful at catching the O(n^2) unindexed query or accidental O(scary) at the app tier caused by a refactor. Or the data migration which wasn’t batched but needs to be. Developer turnover is only accelerating, with predictable results. At this point we are looking into volume snapshots as a pot…

Catch the 90% of stuff that works with your cut down dataset. The other 10% includes things you might only catch not only with O(n^2) queries over a 10TB, but also when under prod load like you say.

Just either gradually roll out those high-risk changes with feature flags, etc. or make sure your monitoring is up-to-scratch so you can catch issues and fix them.

Re: We don’t use a staging environment

#355

Post by nprateem »

myth2018 wrote: post #119

I'm assuming this is not an April Fools' joke, and my comments are targeted at the discussion it sparked here anyway. A flat branching model simplify things, and the strategy they describe surely enables them to ship features to production faster. But the risks I see there: - who decides when a feature is ready to go to production? The programmer who developed them? The automated tests? - features toggleable by a fla…

> - who decides when a feature is ready to go to production? The programmer who developed them? The automated tests?

Exactly. That's the standout claim from the whole article. "We only ship when we're sure code is ready for prod". What, after running a few tests on your laptop? That's a good one :D

Re: We don’t use a staging environment

#356

Post by tharkun__ »

nprateem wrote: post #354
tatersolid wrote: post #351

Earlier quoted context omitted.

We have a cut-down but “production shaped” dataset for fast environment creation but even with code reviews it hasn’t been successful at catching the O(n^2) unindexed query or accidental O(scary) at the app tier caused by a refactor. Or the data migration which wasn’t batched but needs to be. Developer turnover is only accelerating, with predictable results. At this point we are looking into volume snapshots as a pot…

Catch the 90% of stuff that works with your cut down dataset. The other 10% includes things you might only catch not only with O(n^2) queries over a 10TB, but also when under prod load like you say. Just either gradually roll out those high-risk changes with feature flags, etc. or make sure your monitoring is up-to-scratch so you can catch issues and fix them.

Totally seconding the 80/20 rule here. It's probably less frustrating and faster for everyone if you can work on compartmentalizing failures that do slip through or only happen for those two customers with very special data sets that you most probably wouldn't catch in testing either because your test suite won't execise the full 10TB data set anyway (assuming here because we don't know your product)
Post reply on HN