Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Twenty Years of Site Reliability Engineering

sre.google

Post reply on HN
Print view View original

101–110 of 128 posts

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#101

Post by az226 »

benlivengood wrote: post #30

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…

Hyperscalers have several days’ worth of diesel that power generators after the batteries are used up. I’m pretty sure fueling trucks would be routed there should there be a longer power outage than 1-2 days.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#102

Post by nvarsj »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

The fact is many/most SWEs don't have the skillset or interest to do SRE work. While there is a lot of overlap, the work can be quite different between the two areas. SRE basically maps to the sysadmin role of old, which has never really gone away and I don't think it's a product of a "bubble economy".

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#103

Post by foobazgt »

DaiPlusPlus wrote: post #36
dilyevsky wrote: post #10

Earlier quoted context omitted.

Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere. > very broadly applicable. I don't see any "this would only apply at Google" in here. One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod

> Afair Google just ran irc on their corp network which was completely separate from prod I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?

Way back when, for a while, our local (Google) office's internet access ran off the same physical lines as the local prod datacenter traffic. So, any time there was a datacenter traffic outage of any kind, our office was also out. There weren't a lot of outages of that variant, but we knew immediately when one was happening. It's not particularly fun to have all of your access go out concurrently with a prod outage.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#104

Post by NiloCK »

6LLvveMx2koXfwn wrote: post #3

"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?

To think - if it'd been a laptop they would have had to smash it with a hammer.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#105

Post by pinkgolem »

goalieca wrote: post #76
xena wrote: post #53

Earlier quoted context omitted.

IRC servers are single threaded. You have contention at that point.

Some of the bigger irc channels have thousand users and this was in the day of 128MB of ram.

Sure it's possible. But you need to test... For example there was/is a 1k file descriptor limit per process in a lot of distributions..

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#106

Post by jabradoodle »

vsareto wrote: post #49
davedx wrote: post #45

Earlier quoted context omitted.

Eh it’s not the same thing. (I’m very full stack with intermittent devops/sre experience). Full stack means you write code running on back end and front end. 99% of the time the code you write on the FE interfaces with your other code for the BE. It’s pretty coherent and feedback loops are similar. Devops/SRE on the other hand is very different and I agree we shouldn’t expect software developers be mixing in SRE in t…

If you can be good at front and back end and keep up with both of them simultaneously, that's great, but: - you spend more time to keep up with both of those sectors compared to dedicated front or back end positions - you context switch more often than dedicated positions - you spent more time getting good at both of those things - you removed some amount of communication overhead if there were two positions You are…

Is it extra work though, it's not like backend engineers sit around not doing anything because they don't have FE work to do.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#107

Post by crabbone »

namtab00 wrote: post #80
crabbone wrote: post #43

Earlier quoted context omitted.

> much like [...] QA testers have mostly disappeared. Who told you that? QA isn't going anywhere... someone is doing testing, and that someone is a tester. They can be an s/w engineer by training, but as long as they are testing they are a tester. With sysadmins, there are fashion waves, where they keep being called different names like DevOps or SRE. I've not heard of such a thing with testing.

> someone is doing testing, and that someone is a -tester- user excuse me for remembering something surely HN considers a platitude: "everyone has a TEST environment, few are fortunate enough to also have a PROD one"

Well, let me take this seriously for a moment. I believe that companies which don't have dedicated testers today are the same companies which didn't have dedicated testers before.

We really use the language of "users doing the testing" jokingly. No software is written w/o testing, not even very trivial programs would run firs time. So, we just mean that there wasn't enough testing, when we say that.

There is a process, however, that is meant to decrease the number of testers employed. The more testing can be automated, the fewer testers would be necessary... but that hinges on the premise that prior number of testers was somehow sufficient for the amount of testing that was necessary. I believe though that the number of testers hired was a function of budget more than anything else. There's never enough testing, and, in principle, it's hard to see how testing can be exhaustive. So, hopefully, with more automation, it's possible to test more, but, I believe that the number of testers will remain more or less the function of budget.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#108

Post by jeremyjh »

throwawaaarrgh wrote: post #51
benlivengood wrote: post #30

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…

The solution to this has to be done earlier, but it's simple: start a habit of destroying and recreating everything. If you wait to start doing this, it's very painful. If you start doing it at the very beginning, you quickly get used to it, and breaking changes and weird dependencies are caught early. You can even do this with hardware. It changes how you architect things, to deal with shit getting unplugged or rese…

That really doesn't address GP's concerns at all, which are only concerns for the hyperscale cloud providers. Because they use their global infrastructure to turn on a data center that has been down, we don't know if they have a way to restart the entire planet. It is impossible to test this, because that would cause a global outage that could possibly be permanent.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#109

Post by jeremyjh »

Gh0stRAT wrote: post #35
benlivengood wrote: post #30

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz…

Azure has procedures in place to prevent circular dependencies, and regularly exercises them when bringing new regions online. IIRC some of the information about their approach is considered sensitive so I won't elaborate further.

Are you saying they can bring a new data center online without any connectivity to the rest of their infrastructure? GP isn't concerned about turning on one data center, they are concerned about turning them all on at the same time, and that can never be tested.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#110

Post by [deleted] »

riwsky wrote: post #88
jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

God, I feel like such an idiot. All this time I’ve been making fun of Google for having Google Talk, Hangouts, Allo, Duo, Messages, Spaces, Wave, Buzz, Plus, and Meet, I’d never realized that it’s simply a necessary SRE measure at their scale.

[deleted]
Post reply on HN