Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don’t use a staging environment

squeaky.ai

Post reply on HN
Print view View original

231–240 of 357 posts

Re: We don’t use a staging environment

#231

Post by Rapzid »

lupire wrote: post #228
Rapzid wrote: post #218

Earlier quoted context omitted.

Facebook can completely break the user experience for 4.3 million different users each day and each user would only experience one breakage per year. This is pretty common, but not because most employing it have 1.6bn users and 10k engineers; essentially enough scale to throw bodies at problems.

Why would a mistake only affect 4M users and not 400M?

No guarantees, but a feature flag or canary deploy can significantly increase likelihood of impacting a targeted subset of users.

Re: We don’t use a staging environment

#233

Post by SgtBastard »

lupire wrote: post #228
Rapzid wrote: post #218

Earlier quoted context omitted.

Facebook can completely break the user experience for 4.3 million different users each day and each user would only experience one breakage per year. This is pretty common, but not because most employing it have 1.6bn users and 10k engineers; essentially enough scale to throw bodies at problems.

Why would a mistake only affect 4M users and not 400M?

Look at it the other way around - you could have a different outage every single day and as long as that outage only impacted 4.3m users and they were different users each day, it would look like a once-a-year event to the average user.

They’re saying there’s a lot of leeway to break things (in a small way) at scale.

Re: We don’t use a staging environment

#234

Post by thefreeman »

joshuamorton wrote: post #107
productceo wrote: post #26

> We only merge code that is ready to go live. In their perception, is the rest of tech industry gambling in every pull request that some untested code would work in production? I work at a large company. We extensively test code on local machines. Then dev test environments. Then small roll out to just a few data centers in prod bed. Run small scale online flight experiments. Then roll out to the rest of prod bed. A…

Generally speaking yes, I think that if you aren't hiding stuff behind feature flags you're gambling.

just because you have a staging environment doesn't mean you shouldn't use feature flags.

Re: We don’t use a staging environment

#235

Post by catlifeonmars »

withinboredom wrote: post #219
kgeist wrote: post #208

Earlier quoted context omitted.

>Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. Our deployments aren't strictly "synchronous" either. We have thousands of database shards which are all migrated one by one (with some degree of parallelism), and new code is deployed only after all the shards ha…

We don't have a staging environment (for the backend) at work either. However, depending on the size of the tables in-question, a migration might take days. Thus, we usually ask DBA's for a migration days/weeks before any code goes live. There's usually quite a bit of discussion, and sometimes suggestions for an entirely different table with a join and/or application-only (in code, multiple query) join.

Sorry for the silly question, perhaps, but what is the purpose of a db migration? Do schemas in production change that often?

For context, the last couple of services I wrote all have fixed, but implicit schema, (built on key value stores). That is, the DB has no types. So instead, the type system is enforced by the API layer. Any field changes so far are gated via API access and APIs have backwards compatibility contracts with API callers.

I’m not saying that the way I do it currently is “correct” - far from it. I strongly suspect it’s influenced by my lack of familiarity with relational databases.

Re: We don’t use a staging environment

#236

Post by jb3689 »

I don't think you can take infrastructure seriously without a staging environment. For many companies that is fine - they don't have significant infrastructure to maintain (or just don't maintain the infrastructure they have).

I work on a team that maintains our database layer and the lack of a staging environment is incredibly painful. Every test has to be done in production and massive effort needs to be taken to proceed safely. With a staging environment you can be more aggressive and come up with a solid benchmark and test suite to gain confidence rather than having to data collect in prod

Re: We don’t use a staging environment

#237

Post by alex_young »

rock_hard wrote: post #8

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

What about third party integrations? Don’t you need some non-production environment to test them in until both parties are satisfied with the integration and it’s impact on users?

Re: We don’t use a staging environment

#238

Post by jb3689 »

ohmanjjj wrote: post #65

I’ve been shipping software for over two decades, built multiple successful SaaS companies, and have never in my life written a single unit test.

This might as well read that you've been shipping software for over two decades and have never cared about someone introducing a regression

Re: We don’t use a staging environment

#239

Post by withinboredom »

catlifeonmars wrote: post #235
withinboredom wrote: post #219

Earlier quoted context omitted.

We don't have a staging environment (for the backend) at work either. However, depending on the size of the tables in-question, a migration might take days. Thus, we usually ask DBA's for a migration days/weeks before any code goes live. There's usually quite a bit of discussion, and sometimes suggestions for an entirely different table with a join and/or application-only (in code, multiple query) join.

Sorry for the silly question, perhaps, but what is the purpose of a db migration? Do schemas in production change that often? For context, the last couple of services I wrote all have fixed, but implicit schema, (built on key value stores). That is, the DB has no types. So instead, the type system is enforced by the API layer. Any field changes so far are gated via API access and APIs have backwards compatibility con…

A fairly recent example is a couple of tables for users who are “tagged” for marketing purposes (such as we sent them an email and want to display the same messaging in the app). These tags have an expiration date at the tag level but we wanted the expiration date per-user too. This enables marketing to create static tags. This requires a migration to the data so this can be supported.

Schemas don’t change that often, in my experience.

Re: We don’t use a staging environment

#240

Post by pclmulqdq »

drewcoo wrote: post #90
repler wrote: post #86

Earlier quoted context omitted.

Exactly. “Staging never matches Prod” - well why is that? Make it so!!

I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.

I worked at a financial company that had an exact copy of its largest datacenter deployment in a lab for full-system testing. They recorded packets from the exchange at nanosecond precision, and had the equipment to do exact playback of those packets in the lab.

It was an amazing engineering tool. You could test all sorts of end-to-end performance tweaks there, and have great confidence that you were right.

I also worked at a financial company that did not have one of these. The computers ran substantially slower, and lots of end-to-end performance improvements were left on the table and mired in debate. The mock datacenter cost about $10 million, and was worth every penny.

Post reply on HN