An important piece of context missing from the article is the size of their team. LinkedIn shows 0 employees and their about page lists the two cofounders so I assume they have a team of 2. It's odd that the article talks about the problems with large codebases and multiple people working on a codebase when it doesn't look like they have those problems. With only 2 people, of course they can ship like that.
We don’t use a staging environment
191–200 of 357 posts
Re: We don’t use a staging environment
#192> Pre-live environments are never at parity with production
Same with your laptops... and this is only true if you make it that way. Using things like Docker containers eliminates some of the problem with this too.
> There’s always a queue
This has never been a problem for any of the teams I've been on (teams as large as ~80 people). Almost never do they "not want your code on there too". Eventually it's all got to run together anyway.
> Releases are too large
This has nothing to do with how many environments you have, and everything to do with your release practices. We try to do a release per week at a minimum, but have done multiple releases in a single day as well.
> Poor ownership of changes
Code ownership is a bad practice anyway. It allows people to throw their hands up and claim they're not responsible for a given part of the system. A down system is everyone's problem.
> People mistakenly let process replace accountability
Again - nothing to do with your environments here, just bad development practices.
Re: We don’t use a staging environment
#193> We only merge code that is ready to go live
Cool story, but you don't _know_ if its ready until after.
Look, staging environments are not great, for the reasons described. But just killing staging and having done with it isn't the answer either. You need to _know_ when your service is fucked or not performing correctly.
The only way that this kind of deployment is practical _at scale_ is to have comprehensive end-to-end testing constantly running on prod. This was the only real way we could be sure that our service was fully working within acceptable parameters. We ran captured real life queries constantly in a random order, at a random time (caching can give you a false sense of security, go on, ask me how I know)
At no point is monitoring strategy discussed.
Unless you know how your service is supposed to behave, and you can describe that state using metrics, your system isn't monitored. Logging is too shit, slow and expensive to get meaningful near realtime results. Some companies expend billions taming logs into metrics. don't do that, make metrics first.
> You’ll reduce cost and complexity in your infrastructure
I mean possibly, but you'll need to spend a lot more on making sure that your backups work. I have had a rule for a while that all instances must be younger than a month in prod. This means that you should be able to re-build _from scratch_ all instances and datastores. Instances are trivial to rebuild, databases should also be, but often arn't. If you're going to fuck around an find out in prod, then you need good well practised recovery procedures
> If we ever have an issue in production, we always roll forward.
I mean that cute and all, but not being able to back out means that you're fucked, you might not think you're fucked, but that's because you've not been fucked yet.
its like the old addage, there are two states of system admin: Those who are about to have data loss, and those who have had data loss.
Re: We don’t use a staging environment
#194> If we ever have an issue in production, we always roll forward.
What does it mean to roll forward?
Re: We don’t use a staging environment
#195> We only merge code that is ready to go live Cool story, but you don't _know_ if its ready until after. Look, staging environments are not great, for the reasons described. But just killing staging and having done with it isn't the answer either. You need to _know_ when your service is fucked or not performing correctly. The only way that this kind of deployment is practical _at scale_ is to have comprehensive end-t…
All good advice, but do you also have a rule where our DBs have to be less than a month old in prod? Doesn't look very practical if your DB has >100s of TBs
Re: We don’t use a staging environment
#196All of their “problems” with staging are fixable bathwater that doesn't require baby ejection.
I avoid staging for solo projects but it does feel a bit dirty.
For team work or complex solo projects (such as anything commercial) I would never!
On the cloud it is too easy to stage.
To the point where I have teared down and recreated staging environment to save a bit of money at times because it is so easy to bring back.
The article says to me their not using modern devops practices.
It is rare a tech practice “hot take” post is on the money, and this post follows the rule not the exception.
Have a staging environment!
Just the work / thinking / tech debt payoff to make one is worth it for other reasons: including to streamline your deployment processes both human and in code.
Re: We don’t use a staging environment
#197> If we ever have an issue in production, we always roll forward. What does it mean to roll forward?
I believe it means to only push ahead when things break with fixes rather than rolling back to a previously working version.
Re: We don’t use a staging environment
#198Just wondering, what does this phrase mean?
> If we ever have an issue in production, we always roll forward.
Re: We don’t use a staging environment
#199This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states
This is how my current place does it. The only issue we are having is library / dependency updates have a tendency to work perfectly fine locally and then fail in production due to either some minor difference in environment or scale.
It's a problem to the point that we have 5 year old ruby gems which have no listed breaking changes because no one is brave enough to bump them. I had a go at it and caused a major production incident because the datadog gem decided to kill Kubernetes with too many processes.
Re: We don’t use a staging environment
#200> If we ever have an issue in production, we always roll forward. What does it mean to roll forward?
I assume rather than rollback a botched deploy, they solve the bug and do another push?