Earlier quoted context omitted.
> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the…
>The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev enviro…
We don’t use a staging environment
331–340 of 357 posts
Re: We don’t use a staging environment
#332Earlier quoted context omitted.
> They're not arguing that testing or staging environments are bad, they're just saying their organization couldn't manage to get them working. That is exactly what I got from reading this article. Their staging process was poorly set up and they simply abandoned ship. Additionally, I was getting poor software culture vibes.
Indeed, I found the "We only merge code that is ready to go live" part odd. It seems unrelated to the presence of absence of a staging environment. Where I work, we use staging and also only merge code that is ready to go live. Similarly, "Poor ownership of changes" and "People mistakenly let process replace accountability" just don't seem staging-related to me. I've been in environments where people throw code over…
Re: We don’t use a staging environment
#333Earlier quoted context omitted.
The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors. There's a certain amount of "best practices"…
The article doesn't talk about any of that though. The article says staging diffs prod because of: > different hardware, configurations, and software versions The hardware might be hard or expensive to get an exact match for in staging (but also, your stack shouldn't be hyper fragile to hardware changes). The latter two are totally solvable problems
Re: We don’t use a staging environment
#334Earlier quoted context omitted.
Multiply that by the lost hours waiting for compilation, etc and whether you have staging or not doesn’t matter.
Why would I build on a local machine vs running the build on a server in a datacenter? Per your own arguments, server grade hardware is going to compile much faster than any local workstation.
Re: We don’t use a staging environment
#335Earlier quoted context omitted.
> Doesn't look very practical if your DB has >100s of TBs If that's in one shard, then you've got big issues. with larger DBs you need to be practising rolling replacement replicas, because as you scale the chance that one of your shards cocking up approaches 1. Again, it depends on your use case. RDS solves 95% of your problems (barring high scale and expense) If your running your own DBs then you _must_ be replacin…
I work in VFX and we have 1 primary 1 replica for the render farm (MySQL), and another one for an asset system. They both have 100s of TBs many cores and a lot of RAM, we treat them a bit like unicorn machines (they're bare metal), which isn't ideal, but yeah.. our failover and whatnot is to make the primary the replica and vice versa. I cannot imagine reprovisioning it very often, when I worked in startups and used…
Any kind of recovery needed for that sized array takes days, so I feel your pain there.
Re: We don’t use a staging environment
#336This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states
>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables…
Reminiscent of Knight Capital losing $440 million in 45 minutes via feature flags: https://dougseven.com/2014/04/17/knightmare-a-devops-caution...
Re: We don’t use a staging environment
#337This is a pretty weird article. Their "how we do it" section lists: - "We only merge code that is ready to go live" - "We have a flat branching strategy" - "High risk features are always feature flagged" - "Hands-on deployments" (which, from their description, seems to be just a weird way of saying "we have good monitoring and observability tooling") ...absolutely none of which conflict with or replace having a stagi…
FWIW I don't think it is weird at all. Maybe a little short on details of what ready really means for example. While I don't think going completely staging-less makes a lot of sense, going without a shared staging environment is a good thing. It is absolutely awesome to be able to have your own "staging" environment for testing that is independent of everyone else. With the Cloud this is absolutely possible. Shared s…
We regularly see devs write queries or nested loops that work fine on their tiny dev system but explode when run against realistic customer data.
Right now we go through a 12 hour restore then anonymization pass to keep staging up to date. Very painful. Who has a better way?
Re: We don’t use a staging environment
#338Earlier quoted context omitted.
> Otherwise it wouldn't be possible to have migrations that work with both old and new code. Sure you can. Say that you've changed the type of a column in an incompatible way. You can, within a migration that executes as an SQL transaction: 1. rename the original table "out of the way" of the old code 2. add a new column of the new type 3. run an "INSERT ... SELECT ..." to populate the new column from a transformatio…
Not to be rude but this isn't how this works at all. Things like 'run an "INSERT ... SELECT ..."' can't happen at scale due to locking. How they actually do it is super rad: https://www.percona.com/doc/percona-toolkit/3.0/pt-online-sc... tl;dr; They setup a system of triggers (updates,inserts,etc) , copy the data over, then run through all the data in the trigger system. percona developed all these fancy features as…
At scale, you're hopefully not using an RDBMS as a source of truth in the first place; but rather, using it as a CQRS/ES aggregate, downstream of a primary event store (itself likely some combination of a durable message-queue, and archival object storage for compacted event segments.) In that kind of setup, data migrations aren't the job of the RDBMS itself, but rather the job of the CQRS/ES framework — which can simply be taught a new aggregate that computes a single new column, and will start catching that aggregate up and making the data from it available. If your RDBMS is columnar (again, hopefully), you can just begin loading that new column in, no locking on the rest of the table required.
IMHO, the trigger-based approach is a weak approximation of having a pre-normalization primary source. It's fine if you want to avoid rearchitecture, but in a HOLAP system (which is usually inevitable for these sorts of systems when their data architects have tried to focus on simplicity, as this leads to eschewing denormalized secondary representations in OLAP-focused stores) it will cause your write perf to degrade + bottleneck, which is actually the worst thing for locking.
(I should know; I'm dealing with a HOLAP system right now with large numbers of computed indices, append-only OLTP inserts, and random OLAP reads; where the reads hold hundreds of locks each due to partitioning. Any time a write tx stays open for more than a few hundred milliseconds, the whole system degrades due to read locks piling up to the point that the DB begins to choke just on allocating and synchronizing them. The DB is only a few TB large, and the instance it's on has 1TB of memory and as many cores as one can get... but locking is locking.)
Re: We don’t use a staging environment
#339Does anyone know any good resources for testing in production?
Re: We don’t use a staging environment
#340Earlier quoted context omitted.
Multiply that by the lost hours waiting for compilation, etc and whether you have staging or not doesn’t matter.
Why would I build on a local machine vs running the build on a server in a datacenter? Per your own arguments, server grade hardware is going to compile much faster than any local workstation.
Often workstation class processors, like an i9 are the sweet spot, especially with ccache as only one file is being recompiled.
Servers often sacrifice single thread performance for many more cores.