201–210 of 357 posts
> We only merge code that is ready to go live Cool story, but you don't _know_ if its ready until after. Look, staging environments are not great, for the reasons described. But just killing staging and having done with it isn't the answer either. You need to _know_ when your service is fucked or not performing correctly. The only way that this kind of deployment is practical _at scale_ is to have comprehensive end-t…
> they don’t want your changes to interfere with their validation.
Almost like those are issues you want to catch. That's the whole point of continuous integration!
This is good insofar as it forces you to make local development possible. In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally. That leads to people only testing in staging envs, causing staging to constantly break and discouraging automated tests that prevent regression bugs. It also leads to increasing complexity and interconn…
Ehh... once your systems use more than a few pieces of cloud infrastructure / SaaS / PaaS / external dependencies / etc, purely local development of the system is just not possible. There are some (limited) simulators / emulators / etc available and whatnot for some services, but running a full platform that has cloud dependencies on a local machine is often just not possible.
(I can go on a rant about AWS Lambda, and how if they'd used a standardized interface like FastCGI it would make local testing trivial, but they won't do that because they need vendor lock-in...)
Imagine writing this entire blog post and being completely wrong about every topic you discuss. This is the most amateur content I've seen make it to the front page, let alone top post.
> Pre-live environments are never at parity with production Same with your laptops... and this is only true if you make it that way. Using things like Docker containers eliminates some of the problem with this too. > There’s always a queue This has never been a problem for any of the teams I've been on (teams as large as ~80 people). Almost never do they "not want your code on there too". Eventually it's all got to r…
Agreed with a lot of what you said up until this - this is, frankly, just completely wrong. If nobody has any ownership over anything, nobody is compelled to fix anything - I've experienced this first-hand on multiple occasions.
There have also been several studies done to refute your point - higher ownership correlates with higher quality. A particularly well-known one is from Microsoft, which had a follow up study later that attempted to refute the original findings but failed to do so. Granted, these were conducted from the perspective of code quality, but it is trivial to apply the findings to other scenarios that demand accountability.
[1] https://www.microsoft.com/en-us/research/wp-content/uploads/...
[2] https://www.microsoft.com/en-us/research/wp-content/uploads/...
Whoever sold you on the idea that ownership of _any and all kinds_ is bad would likely rather you be a replaceable cog than someone of free thought. I don't know about you, but I take pride in the things I'm responsible for. Most people are that way. I also don't give two shits about anything that I don't own, because there's not enough time in the day for everyone to care about everything. This is why we have teams in the first place.
There is a mile of difference between toxic and productive ownership - Gatekeepers are bad, custodians are good.
Earlier quoted context omitted.
Exactly. “Staging never matches Prod” - well why is that? Make it so!!
I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.
You set the CI/CD pipeline to enforce that deploys happen to staging, and then happen to production. That's it. It's not hard.
Earlier quoted context omitted.
The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors. There's a certain amount of "best practices"…
The article doesn't talk about any of that though. The article says staging diffs prod because of: > different hardware, configurations, and software versions The hardware might be hard or expensive to get an exact match for in staging (but also, your stack shouldn't be hyper fragile to hardware changes). The latter two are totally solvable problems
Earlier quoted context omitted.
>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables…
> Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. I didn't work on FB databases while I was employed by them, but when you've got a lo…
Our deployments aren't strictly "synchronous" either. We have thousands of database shards which are all migrated one by one (with some degree of parallelism), and new code is deployed only after all the shards have migrated. So there's a large window (sometimes up to an hour) when some shards see the new schema and others see the old schema (while still running old code). It's one click of a button, however, and one logical release, we don't split it into separate releases (so I view them as "automatic"). The problem still stays, though, that you can only guard code with feature flags, migrations can't be conditionally disabled. With this setup, if a poorly tested migration goes awry, it's even more difficult to rollback, because it will take another hour to roll back all the shards.
Earlier quoted context omitted.
Ehh... once your systems use more than a few pieces of cloud infrastructure / SaaS / PaaS / external dependencies / etc, purely local development of the system is just not possible. There are some (limited) simulators / emulators / etc available and whatnot for some services, but running a full platform that has cloud dependencies on a local machine is often just not possible.
The answer (IMHO) is to not use services that make it impossible to develop locally, unless you can trivially mock them; the benefits of such services aren't worth it if they result in a system that is inherently untestable with an environment that's inherently unreproducible. (I can go on a rant about AWS Lambda, and how if they'd used a standardized interface like FastCGI it would make local testing trivial, but th…
Earlier quoted context omitted.
>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables…
> Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. I didn't work on FB databases while I was employed by them, but when you've got a lo…