Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Twenty Years of Site Reliability Engineering

sre.google

Post reply on HN
Print view View original

111–120 of 128 posts

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#111

Post by Guvante »

kortilla wrote: post #94
fragmede wrote: post #57

Earlier quoted context omitted.

The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my tea…

It’s still batshit insane to me that Google Fi had any infrastructure overlap with Gmail. When I was there these grand unified base layer systems everything else was built on top of were a point of pride. From a risk perspective though it was sheer stupidity in retrospect. As Google continues to atrophy and suffers attrition of the original people that built those systems, the probability creeps up more and more that…

At my not Google job we talk about "what happens if a meteor hits a DC".

We agree that that is so rare that as long as there are buttons we can push to recover after a reasonable timeframe that is an acceptable risk, we don't need a fully automatic way to recover from that.

However your SRE teams needs a way to recover without intervention which is why there is talk of backups.

BTW even using different cloud providers isn't enough to avoid a DC outage necessarily. No amount of redundancy can protect you from it beyond a ton of services which intentionally slice off access to the DC leading to the risk of that happening accidentally which is its own risk.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#112

Post by fatnoah »

fragmede wrote: post #57
jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my tea…

> The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent.

Not sure this is an "only at google" thing. In a past life, I ran Engineering for an App/SMS messaging application. We definitely used an IRC channel as well since the nature of our outages would mean messaging channels could be down. It's also why we didn't rely soley on SMS for our on-call alerting.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#113

Post by seedless-sensat »

I find it interesting that this reflection didn't mention SLI/SLO/error budgets, which Google SRE has championed for a long time.

My impression is that they're nice in theory, but less useful in practice. I'm yet to see an error budget effectively inform eng decision making.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#114

Post by jabroni_salad »

seedless-sensat wrote: post #113

I find it interesting that this reflection didn't mention SLI/SLO/error budgets, which Google SRE has championed for a long time. My impression is that they're nice in theory, but less useful in practice. I'm yet to see an error budget effectively inform eng decision making.

Error budgets are to control the workload of the guy who is holding the oncall pager, who otherwise has no say over his or her situation. In recent years companies have shifted to 'you build it you own it' and the infra has been abstracted to the point that the SWE can own the entire thing.

Error budgets also only matter if you either give a shit about your guys or have to pay them for that oncall time. Plenty of employers are happy so say 'salary is exempt, suck it up lol' so errors are effectively free.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#115

Post by derefr »

zdragnar wrote: post #75
derefr wrote: post #56

Earlier quoted context omitted.

But modern approaches to dev require the SWEs to understand and model the operation of their software, and in fact program in terms of it — “writing infrastructure” rather than just code. Lambda functions, for example: you have to understand their performance and scalability characteristics — in turn requiring knowledge of things like the latency added by crossing the boundary between a managed shared service cluster…

That is barely tip-toeing across the very edges of SRE land.

Alright, how about expecting devs to repackage their entire until-that-point-SaaS stack into an "appliance" (Kubernetes Helm chart), containing SWE-written resource manifests that define the application's scaling characteristics across arbitrarily-shaped k8s clusters they won't get to see in advance, using only node taints; memory limits for layers of their stack they've never even seen run full-bore before; health checks that multiplex back up to a central monitoring platform; safely-revertible multiphase upgrade rollout behavior that never decreases availability; and so forth;

...and then those same devs being expected to directly debug the behavior of this "appliance" in a client environment (think: someone consuming the "appliance" through the Amazon Marketplace, where this launches the workload into an EKS cluster in the customer's own VPC, with the customer in control of defining that cluster's node pools);

...where this can involve, for example, figuring out that a seemingly-innocent bounded-size Redis cache deployment, needs 10x its steady-state memory, when booting from a persisted AOF file... for some godforsaken reason.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#116

Post by gen220 »

oceanplexian wrote: post #78
gen220 wrote: post #29

Earlier quoted context omitted.

I imagine the threshold is something like 1 SRE for every $1mm of high-margin revenue you can link to guaranteeing the 2nd "9" of $product availability/reliability.

That’s sort of ridiculous. A mid-level SRE easily costs a quarter of that. And a company like Apple would then have 80,000 SREs? Lol no.

I think you've perhaps misread my post?

It's marginal revenue attributable to a high-performing SRE (i.e. an SRE who would be able to elevate a product they're supporting from 90.0% availability to 99.0% availability.

It's actually a pretty high bar, because there aren't that many products for which the that segment of availability translates to >$1mm in marginal revenue. $1mm is a ballpark figure, but I think it's the right order of magnitude (i.e. the true number might be $5mm).

Expanding on another point in the original post: decision varies with the profitability of that marginal revenue. For example, it's basically pure profit for Google, Amazon or Netflix – accordingly, it makes sense that they'd have many people who focus exclusively on performance and availability, to make sure they aren't leaving that revenue on the ground.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#117

Post by nonameiguess »

fragmede wrote: post #57
jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my tea…

I wouldn't say this is a Google only problem. Really, all of these problems of redundancy and recovery have been best considered by militaries well before private companies started to have sufficiently global operations and communications networks that they started thinking about it, too. In the Army Recon Course, they taught us to bounce signals off the ionosphere using hand-held radios if we needed to communicate with an otherwise cut-off remote unit too far past the earth's curvature horizon to get line of sight. And, of course, every unit is drilled in contingency plans to the point that it can operate independently as necessary even when all communication gets cut off and continue to push the overall mission forward even in the absence of further coordination, which, for what it's worth, obviously does present the potential for problems of its own when the mission changes and a disconnected unit doesn't know it, as well explored in Doctor Strangelove.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#118

Post by theptip »

fatnoah wrote: post #112
fragmede wrote: post #57

Earlier quoted context omitted.

The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my tea…

> The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. Not sure this is an "only at google" thing. In a past life, I ran Engineering for an App/SMS messaging application. We definitely used an IRC channel as well since the nature of our outages would mean messaging channels could be down. It's also why we didn't rely soley on SMS…

I think they meant it’s only at Google that Gmail would not be an uncorrelated email etc.

Though I think Microsoft probably has the equivalent problem with Exchange, Teams, etc.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#119

Post by sumedh »

teddyh wrote: post #93
xyst wrote: post #34

Off topic: TIL Google has its own TLD (.google)

From what I can tell, Google owns at least eleven TLDs, just for themselves: • .android • .cal • .chrome • .gbiz • .gle • .gmail • .goog • .google • .play • .prod • .youtube Google also owns 22 generic domains: • .app • .boo • .channel • .dad • .day • .dev • .eat • .esq • .fly • .foo • .hangout • .here • .how • .ing • .meme • .mov • .new • .nexus • .page • .prof • .search • .zip

How did Apple allow Google to take over .app

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#120

Post by toast0 »

Racing0461 wrote: post #90
xyst wrote: post #34

Off topic: TIL Google has its own TLD (.google)

For a sr sde yearly salary, you can own one too. The application process is nevertheless, "may issue".

Are they accepting applications again, or do you also need a time machine?
Post reply on HN