Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Twenty Years of Site Reliability Engineering

sre.google

Post reply on HN
Print view View original

51–60 of 128 posts

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#51

Post by throwawaaarrgh »

benlivengood wrote: post #30

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…

The solution to this has to be done earlier, but it's simple: start a habit of destroying and recreating everything. If you wait to start doing this, it's very painful. If you start doing it at the very beginning, you quickly get used to it, and breaking changes and weird dependencies are caught early.

You can even do this with hardware. It changes how you architect things, to deal with shit getting unplugged or reset. You end up requiring more automation, version control and change management, which speeds up and simplifies overall work, in addition to preventing and quickly fixing outages. It's a big culture shift.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#52

Post by throwawaaarrgh »

codemac wrote: post #9
6LLvveMx2koXfwn wrote: post #3

"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?

It always cracks me up how Google is simultaneously the most web-based company in the world, but their internal political landscape was (Infra, Search, Ads) > everything else. This leads to infra swe writing stupid CLIs all day, rather than having any literal buttons. Things were changing a lot by the time I left though. I do think Google should be more open about their internal outages. This one in particular was ve…

Every large enterprise is internally a tire fire.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#53

Post by xena »

londons_explore wrote: post #17
gaogao wrote: post #16

Earlier quoted context omitted.

Also make sure your backup channel can scale. Getting a flood of 10,000+ folks over to a dinky IRC server can knock it over easy. Throttling new joinees isn't a panacea either, since there might be someone critical to get on a channel which throttling can complicate.

Maybe I'm naive, but I would imagine that any raspberry pi could run an IRC server with 10,000 users... Surely 10,000 Users each on average receiving perhaps five 50 byte messages per second (that's a very busy chat room!) is a total bandwidth of 2.5 megabytes per second. And the CPU time to shuffle messages from one TCP connection to another, or encrypt/decrypt 2.5 megabytes per second should be small. There is no c…

IRC servers are single threaded. You have contention at that point.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#54

Post by donalhunt »

gen220 wrote: post #29
fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

I imagine the threshold is something like 1 SRE for every $1mm of high-margin revenue you can link to guaranteeing the 2nd "9" of $product availability/reliability.

I believe that is indeed a good guide for when it makes sense to have a SRE team supporting a service or product (with the caveat that the number probably isn't $1MM).

There are also good patterns for ensuring you actually have adequate SRE coverage for what the business needs. 2 x 6ppl teams geo-graphically dispersed doing 7x12 shifts works pretty well (not cheap). You can do it with less but you run into more challenges when individuals leave / get burnt out / etc.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#55

Post by strangattractor »

jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

Brings back memories.

I started getting phone calls at 5AM regarding servers going down. I immediately went to email to connect with other members of the team and found my inbox full of thousands of emails from the monitoring service notifying me of the outage. Those were days:)

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#56

Post by derefr »

davedx wrote: post #47
kccqzy wrote: post #21

Earlier quoted context omitted.

SRE is not a byproduct of a bubble economy. I believe Google has had SREs since the very beginning. But still I think the rest of the point still stands. These days with devops the skill set needed for devs have indeed expanded to have significant overlap with SREs. I expect companies to downsize their SRE teams and distribute responsibilities to devs. A second major reason is automation. If you read the linked site…

> “These days with devops the skill set needed for devs have indeed expanded to have significant overlap with SREs” Respectfully disagree on this. SRE is a huge complex realm unto itself. Just understanding how all the cloud components and environments and role systems work together is multiple training courses, let alone how to reliably deploy and run in them.

But modern approaches to dev require the SWEs to understand and model the operation of their software, and in fact program in terms of it — “writing infrastructure” rather than just code.

Lambda functions, for example: you have to understand their performance and scalability characteristics — in turn requiring knowledge of things like the latency added by crossing the boundary between a managed shared service cluster and a VPC — in order to understand how and where to factor things into individual deployable functions.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#57

Post by fragmede »

jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my team's system is down, it's not improbable that Google Meet (along with Google.com) was down because of it.

We needed to have extra layers of backup communications because the first three layers systems were all also Google properties. That is to say, email wouldn't work because Gmail is Google, our work cell phones wouldn't work because they're Google Fi, and my home's ISP was Google Fiber/Webpass.

All of which is to confirm that, yes, Google has a backup IRC server for communication. I won't say where, but it's explicitly totally off Google infrastructure for that very reason.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#58

Post by masto »

tiddo_langerak wrote: post #11

I'm curious how people approach big red buttons and intentional graceful degradation in practice, and especially how to ensure that these work when the system is experiencing problems. E.g. do you use db-based "feature flags"? What do you do then if the DB itself is overloaded, or the API through which you access the DB? Or do you use static startup flags (e.g. env variables)? How do you ensure these get rolled out q…

It depends.

To make up an example that doesn't depend on any of those things: imagine that I've added a new feature to Hacker News that allows users to display profile pictures next to their comments. Of course, we have built everything around microservices, so this is implemented by the frontend page generator making a call to the profile service, which does a lookup and responds with the image location. As part of the launch plan, I document the "big red button" procedure to follow if my new component is overloading the profile service or image repository: run this command to rate-limit my service's outgoing requests at the network layer (probably to 0 in an emergency). It will fail its lookups and the page generator is designed to gracefully degrade by continuing to render the comment text, sans profile photo.

(Before anyone hits send on that "what a stupid way to do X" reply, please note that this is not an actual design doc, I'm not giving advice on how to build anything, it's just a crayon drawing to illustrate a point)

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#59

Post by jeffbee »

benlivengood wrote: post #30

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…

When I was in Google SRE we had monitoring and enforcement of permitted and forbidden RPC peers, such that a system that attempted to use another system would fail or send alerts. This was useful at the top of the stack to keep track of dependencies silently added by library authors, and at the low levels to ensure the things at the bottom of the stack were really at the bottom. We also did virtual automated cluster turn-up and turn-down, to make sure our documented procedures did not get out of date, and in my 6 years in SRE I saw that procedure fall from 90 days to under an hour. We also regularly exercised the scratch restarts of things like global encryption key management, which involves a physical object. The annual DiRT exercise also tried to make sure that no person, team, or office was necessary to the continuing function of systems.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#60

Post by jeffbee »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

If you think of an SRE as an expensive sysadmin then yes, you should absolutely scratch that entire org. SRE, by Google's definition, is supposed to contain software engineers with deep systems expertise, not some kind of less-qualified SWEs.
Post reply on HN