Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Twenty Years of Site Reliability Engineering

sre.google

Post reply on HN
Print view View original

91–100 of 128 posts

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#91

Post by slyall »

crabbone wrote: post #43
fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

> much like [...] QA testers have mostly disappeared. Who told you that? QA isn't going anywhere... someone is doing testing, and that someone is a tester. They can be an s/w engineer by training, but as long as they are testing they are a tester. With sysadmins, there are fashion waves, where they keep being called different names like DevOps or SRE. I've not heard of such a thing with testing.

> With sysadmins, there are fashion waves, where they keep being called different names like DevOps or SRE.

I don't think the name change really originated with Sysadmins. Basically these new titles were created (with narrow definitions) and then other companies said "We are cool like Google, we have SREs now, no Sysadmins" so all the jobs had new titles.

Source: Me and my last 4 jobs ( Sysadmin -> Devops Engineer -> Infrastructure Developer -> SRE ) which are all basically the same thing

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#92

Post by Jensson »

riwsky wrote: post #88
jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

God, I feel like such an idiot. All this time I’ve been making fun of Google for having Google Talk, Hangouts, Allo, Duo, Messages, Spaces, Wave, Buzz, Plus, and Meet, I’d never realized that it’s simply a necessary SRE measure at their scale.

Imagine if they accidentally deprecated their last messaging service? Can't let that happen so better build some extra just in case.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#93

Post by teddyh »

xyst wrote: post #34

Off topic: TIL Google has its own TLD (.google)

From what I can tell, Google owns at least eleven TLDs, just for themselves:

• .android

• .cal

• .chrome

• .gbiz

• .gle

• .gmail

• .goog

• .google

• .play

• .prod

• .youtube

Google also owns 22 generic domains:

• .app

• .boo

• .channel

• .dad

• .day

• .dev

• .eat

• .esq

• .fly

• .foo

• .hangout

• .here

• .how

• .ing

• .meme

• .mov

• .new

• .nexus

• .page

• .prof

• .search

• .zip

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#94

Post by kortilla »

fragmede wrote: post #57
jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my tea…

It’s still batshit insane to me that Google Fi had any infrastructure overlap with Gmail. When I was there these grand unified base layer systems everything else was built on top of were a point of pride. From a risk perspective though it was sheer stupidity in retrospect.

As Google continues to atrophy and suffers attrition of the original people that built those systems, the probability creeps up more and more that someone will one day cause a catastrophic global outage of systems that should share absolutely no dependencies but do because “it scales”.

If Google made auto pilot for airplanes, it would all be on a centralized SaaS with white papers published in academic conferences about the elaborate systems designed to ensure server crashes don’t impact it. Nobody will ever complain about how all of those guarantees depend on chubby, big table, etc.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#95

Post by pulsetracker »

jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

For small startups, WhatsApp group or phone call can act as a backup.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#96

Post by bravetraveler »

zdragnar wrote: post #75
derefr wrote: post #56

Earlier quoted context omitted.

But modern approaches to dev require the SWEs to understand and model the operation of their software, and in fact program in terms of it — “writing infrastructure” rather than just code. Lambda functions, for example: you have to understand their performance and scalability characteristics — in turn requiring knowledge of things like the latency added by crossing the boundary between a managed shared service cluster…

That is barely tip-toeing across the very edges of SRE land.

Yea, this is buying and using toys. Need to go down a few layers of abstraction

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#97

Post by kevan »

jeffbee wrote: post #61
6LLvveMx2koXfwn wrote: post #3

"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?

It's a logical consequence of the "zero trust" network. If an engineer's workstation can make RPCs to production systems, and that engineer is properly entitled to assume some privileged role, then there's no difference between running the automation in prod and running it on your workstation. Even at huge scales, shell tools plus RPC client CLIs can contact every machine in the world pretty promptly.

There's still differences. If you're running it in prod then the functionality has at least gone through code review and you have higher confidence what's running is what you think it is. If you run things from personal boxes there's always the risk of them not having the latest code, having made a local change and not checking it in, or the worst case of a bad actor doing whatever they want with the privileged role. But if code review isn't required or engineers have unrestricted SSH access to prod hosts then it's pretty much equivalent.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#98

Post by tap-snap-or-nap »

Smaug123 wrote: post #28

For much much more on this, I'm most of the way through Google's book _Building Secure and Reliable Systems_, which is a proper textbook (not light reading). It's a pretty interesting book! A lot of what it says is just common sense, but as the saying goes, "common sense" is an oxymoron; it's felt useful to have refreshed my knowledge of the whole thing at once.

For those who want to read it https://google.github.io/building-secure-and-reliable-system...

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#99

Post by Too »

benlivengood wrote: post #30

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…

Not just cold starts. Similar struggles if your Infra-as-code deployments depend on your CI/CD pipelines, conveniently running in the same environment as the one having an outage that you need to push new configs to.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#100

Post by pests »

londons_explore wrote: post #17
gaogao wrote: post #16

Earlier quoted context omitted.

Also make sure your backup channel can scale. Getting a flood of 10,000+ folks over to a dinky IRC server can knock it over easy. Throttling new joinees isn't a panacea either, since there might be someone critical to get on a channel which throttling can complicate.

Maybe I'm naive, but I would imagine that any raspberry pi could run an IRC server with 10,000 users... Surely 10,000 Users each on average receiving perhaps five 50 byte messages per second (that's a very busy chat room!) is a total bandwidth of 2.5 megabytes per second. And the CPU time to shuffle messages from one TCP connection to another, or encrypt/decrypt 2.5 megabytes per second should be small. There is no c…

Let's not forget IRC channels can be split between servers too - don't wanna complicate the backup system too much but this is in the original design of IRC.
Post reply on HN