Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Twenty Years of Site Reliability Engineering

sre.google

Post reply on HN
Print view View original

81–90 of 128 posts

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#81

Post by ahmedfromtunis »

throwawaaarrgh wrote: post #51
benlivengood wrote: post #30

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…

The solution to this has to be done earlier, but it's simple: start a habit of destroying and recreating everything. If you wait to start doing this, it's very painful. If you start doing it at the very beginning, you quickly get used to it, and breaking changes and weird dependencies are caught early. You can even do this with hardware. It changes how you architect things, to deal with shit getting unplugged or rese…

My SaaS app is small even compared to some desktop apps. But at least once a year, I try to reboot it from scratch.

Of course this is way easier for a nano-scale app, but I love the feeling of knowing that it can be started from any server in the world in less than 10 minutes (including copying the data).

I also make sure there are 0 errors even with a clean slate database.

For some reason I can't understand, this gives me some joy.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#82

Post by opportune »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

Even Google is doing this now.

I think it’s simply swapping one set of trade offs for another. With dedicated SREs you have true specialists in production operations and their accompanying systems (tooling, alerting, etc) with a clear mandate and ownership of outcomes; but they don’t necessarily have full ownership of what they’re keeping running, and that can cause organizational problems (we want to launch X, SRE says no, or vice versa) and make it so non-SREs take no ownership over their hard-to-support code.

Conversely you can have Eng teams without SREs and most of those organizational/social problems, at the cost of production reliability being only one of many priorities.

I think what’s really happening is that a lot of companies are deciding they don’t care about reliability very much as a business outcome, especially when it comes at the expense (at least in opportunity cost) of less features.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#83

Post by cortesoft »

jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

Yeah, I work for a CDN and we have to make sure our status page provider is not one of our customers.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#84

Post by 8040 »

I would like to take this moment to really highlight "Recovery mechanisms should be fully tested before an emergency". As the unexpected SRE at Google who became known by entire company for using a double negative incorrectly, it is something very important to do right away.

For those Googlers curious, you can search my username internally for how I generated more impact then could be measured.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#85

Post by _boffin_ »

8040 wrote: post #84

I would like to take this moment to really highlight "Recovery mechanisms should be fully tested before an emergency". As the unexpected SRE at Google who became known by entire company for using a double negative incorrectly, it is something very important to do right away. For those Googlers curious, you can search my username internally for how I generated more impact then could be measured.

Possible to give more insightful details?

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#86

Post by VirusNewbie »

oceanplexian wrote: post #77
VirusNewbie wrote: post #62

Earlier quoted context omitted.

In my G SRE interview, I had to do the same rigorous Software Engineering algorithms rounds as well as show deep distributed systems knowledge in designing highly available systems.

If by rigorous algorithms you mean, spend a month memorizing a few dozen leetcode problems then sure, I’ll agree that is sadly the state of SRE interviews at FAANG.

I interviewed at multiple FAANGs and not one of the questions they asked was on leetcode or hackerrank. (I searched afterwords).

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#87

Post by plygrnd »

benlivengood wrote: post #30

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…

AWS learned this lesson in 2017 when S3 fell over. Lots of internal changes after that.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#88

Post by riwsky »

jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

God, I feel like such an idiot. All this time I’ve been making fun of Google for having Google Talk, Hangouts, Allo, Duo, Messages, Spaces, Wave, Buzz, Plus, and Meet, I’d never realized that it’s simply a necessary SRE measure at their scale.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#89

Post by js2 »

> Automate your mitigations

Think long and hard about this one. Multiple times in my three-decade career I've seen automated mitigations make the problem worst. So really consider whether self-healing is something you need.

I built my company's in-house mobile crash reporting solution in 2014. Part of the backend has had one server running Redis as a single point of failure. The failover process is only semi-automated. A human has to initiate it after confirming alerts about it being down are valid. There's also no real financial cost to it going down - at worst my company's mobile app developers are inconvenienced for a bit.

In the decade the system has been operational I can count on two fingers the number of times I've had to failover.

Despite this system having no SLA it's had better uptime than much more critical internal systems.

Conversely:

https://github.blog/2023-05-16-addressing-githubs-recent-ava...

https://github.blog/2018-10-30-oct21-post-incident-analysis/

https://www.datacenterknowledge.com/archives/2012/12/27/gith...

To be fair, GitHub operates at a much larger scale. My point is only that redundancy and automated mitigations add complexity and are almost by definition rarely tested and operate under unforeseen circumstances.

So really, consider your SLA and the cost of an outage and balance that against the complexity you'll add by guarding against an outage.

I think my first introduction to this was circa 1998 when I had a pair of NetApps clustered into an HA configuration and one of them failed and caused the other to corrupt all its disks. Fun times. A similar thing happened around the same time with a pair of Cisco PIX firewalls. I've been leery of HA and automated failover/mitigations ever since.

Post reply on HN