161–170 of 357 posts
Staging, tests, previews and even running code locally is for people who make mistakes. It's dumb and a total waste of time if you don't make any mistakes. No testing at all, that's what I call optimizing for success! On a more serious note: Sometimes staging is the same as local, and in those situations there is very limited use for staging.
Earlier quoted context omitted.
> If you are saying you don't have a staging environment, what you are really saying is that your company doesn't have any QA process. Come on - this is nonsense. Feature flags for example?
Feature flag systems don’t magically prevent a new feature from causing a bug for other existing features, or even taking the whole site down.
Earlier quoted context omitted.
> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the…
>The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev enviro…
This is definitely best practice, but it's not strictly necessary if a small amount of downtime is acceptable. We only have customers in one timezone and minimal traffic overnight, so we have quite a lot of leeway with this. Frankly even during business hours small amounts of downtime (e.g. 5 minutes) would be well tolerated: it's a lot better than most of the other services they are used to using anyway.
By doing this, I can run a service locally but connect it to the production pubsub server and then see how it effects the system if I publish events to it locally.
I could also subscribe to events and see real production events hitting my local machine.
This made me laugh.
- Everyone has a testing environment. Some people are lucky enough that they have a separate one for running production
[INSERT ADDITIONAL JOKES HERE]
- I don't always test my code but when I do, it's in production. - Everyone has a testing environment. Some people are lucky enough that they have a separate one for running production [INSERT ADDITIONAL JOKES HERE]
This is good insofar as it forces you to make local development possible. In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally. That leads to people only testing in staging envs, causing staging to constantly break and discouraging automated tests that prevent regression bugs. It also leads to increasing complexity and interconn…
At one time this was a huge blocker for our productivity. Access to a reliable test environment was only possible by way of a specific customer's production environment. The vendor does maintain a shared 3rd party integration test system, but its so far away from a realistic customer configuration that any result from that environment is more distracting than helpful.
In order to get this sort of thing out of the way, we wrote a simulator for the vendor's system which approximates behavior across 3-4 of our customer's live configurations. Its a totally fake piece of shit, but its a consistent one. Our simulated environment testing will get us about 90% of the way there now. There are still things we simply have to test in customer prod though.
When they lose all their most important customers’ data because the feature flags got too confusing… they can take this same article and say: “BECAUSE WE xxxx that led to YYYY.
In future we will use a Staging or UAT environment to mitigate against YYYY and avoid xxxx”
Saving time on authoring a Post Mortem by pre-describing your folly seems like an odd way to spend precious dev time
This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states
>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables…
Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. I didn't work on FB databases while I was employed by them, but when you've got a lot of frontends and a lot of sharded databases, you don't have much choice; if your schema is changing, you've got to have a multiphased push:
a) push frontend that can deal with either schema
b) migrate schema
c) push frontend that uses new schema for new feature (with the understanding that the old frontend code will be running on some nodes) --- this part could be feature flagged
d) data cleanup if necessary
e) push code that can safely assume all frontends are new feature aware and all rows are new feature ready
IMHO, this multiphase push is really needed regardless of scale, but if you're small, you can cross your fingers and hope. Or if you're willing to take downtime, you can bring down the service, make the database changes without concurrent access, and bring the service back with code assuming the changes; most people don't like downtime though.