I’ve been shipping software for over two decades, built multiple successful SaaS companies, and have never in my life written a single unit test.
I like hearing about different heterodox approaches.
We don’t use a staging environment
281–290 of 357 posts
Re: We don’t use a staging environment
#282When I started at Twitter, a guy was proposing building a staging environment. At that point it would have cost about $2M, so it was within the realm of conceivable. I was a pretty immediate “no”, for all sorts of reasons. Keeping the data in reasonable shape would have been a big project all by itself, getting reasonable load on it would have been another, and then of course it’s a large environment you need to be on call for but that doesn’t have the priority production has. It’s just an all around bad idea.
Re: We don’t use a staging environment
#283Without a staging environment, how do you test that large scale database migrations work as intended? I wouldn't feel at all comfortable shipping changes like that which have only been tested on laptops.
How do you define a large scale database migration? If you're just updating data or schema, that can be done locally via integration test. No need for a separate environment.
You have a table with 200m rows in it.
You want to me extract one of the columns in that table out to a new, separate table - a job that will take several hours to complete.
You want to do this without any visible downtime or breakage to your end-users - likely with some kind of complex dual-write and/or dual/read mechanism during that operation.
Re: We don’t use a staging environment
#284Earlier quoted context omitted.
> Doesn't look very practical if your DB has >100s of TBs If that's in one shard, then you've got big issues. with larger DBs you need to be practising rolling replacement replicas, because as you scale the chance that one of your shards cocking up approaches 1. Again, it depends on your use case. RDS solves 95% of your problems (barring high scale and expense) If your running your own DBs then you _must_ be replacin…
I work in VFX and we have 1 primary 1 replica for the render farm (MySQL), and another one for an asset system. They both have 100s of TBs many cores and a lot of RAM, we treat them a bit like unicorn machines (they're bare metal), which isn't ideal, but yeah.. our failover and whatnot is to make the primary the replica and vice versa. I cannot imagine reprovisioning it very often, when I worked in startups and used…
Why are those so big? Are you storing binary data (assets) within the tables? Or it just has old historical records that were never "warehoused"?
Re: We don’t use a staging environment
#285Disclaimer: I worked for a major feature flagging company, but these opinions are my own. This article makes a lot of valid points regarding staging environments, but their reasoning to not use them is dubious. None of their reasons are good enough to take staging environments out of the equation. I'd be willing to be that the likelihood of anyone merging code that isn't ready to go live is close to zero. You still n…
How do you feature flag a refactor?
A refactor is not a feature. Same way a bugfix isn't.
Re: We don’t use a staging environment
#286I don't think you can take infrastructure seriously without a staging environment. For many companies that is fine - they don't have significant infrastructure to maintain (or just don't maintain the infrastructure they have). I work on a team that maintains our database layer and the lack of a staging environment is incredibly painful. Every test has to be done in production and massive effort needs to be taken to p…
imho this only applies for teams with average or below average devs
Which part?
Re: We don’t use a staging environment
#287This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing o…
The parity part is especially confusing to me, since they go on to test locally. So staging isn't close enough to production, but local is...?
Re: We don’t use a staging environment
#288Earlier quoted context omitted.
Exactly. “Staging never matches Prod” - well why is that? Make it so!!
I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.
[deleted]
Re: We don’t use a staging environment
#289Earlier quoted context omitted.
I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.
> I have never ever even heard of a place where that was possible. You set the CI/CD pipeline to enforce that deploys happen to staging, and then happen to production. That's it. It's not hard.
Tech debt can definitely accrue that makes it difficult. For instance recently I had an odd difference between prod and staging, and when I looked into it I realized there was a legacy behavior for old customers, and our testing users in different environments were on different sides of the cutoff.
Naturally that's a nasty tech debt and we should find a way to clean it up, but it was a pragmatic solution to a business problem that occurred before any of my team mates started working there, and it meant we could continue to serve our customers as we made a transition. These things happen.
Re: We don’t use a staging environment
#290Earlier quoted context omitted.
I work in VFX and we have 1 primary 1 replica for the render farm (MySQL), and another one for an asset system. They both have 100s of TBs many cores and a lot of RAM, we treat them a bit like unicorn machines (they're bare metal), which isn't ideal, but yeah.. our failover and whatnot is to make the primary the replica and vice versa. I cannot imagine reprovisioning it very often, when I worked in startups and used…
Why are those so big? Are you storing binary data (assets) within the tables? Or it just has old historical records that were never "warehoused"?
We run a lot of workflows in our systems, we keep a rolling window of 3 months for the farm data, the rest goes to a warehouse system. For the asset we keep for a show duration and we have many shows being worked from many studios. We don’t have binary data in them