Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Twenty Years of Site Reliability Engineering

sre.google

Post reply on HN
Print view View original

71–80 of 128 posts

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#71

Post by l9i »

jedberg wrote: post #7

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…

> And I think Google has a backup IRC server on AWS, but that might just apocryphal.

They (>1) do exist but not on AWS or any other major cloud provider.

(Or at least that was the case two years ago when I still worked there.)

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#72

Post by shanemhansen »

DaiPlusPlus wrote: post #36
dilyevsky wrote: post #10

Earlier quoted context omitted.

Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere. > very broadly applicable. I don't see any "this would only apply at Google" in here. One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod

> Afair Google just ran irc on their corp network which was completely separate from prod I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?

They do. But I'd say most employees go their whole career without needing to do anything that requires a VPN.

It's basically all web based access through what is, at the end of the day, a http proxy.

SREs need to be ready for stuff like "hey, what if the big proxy we all use to access internal resources is down?".

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#73

Post by icedchai »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

SysAdmins didn't disappear, they just learned some cloud stuff and changed titles. We call them "DevOps Engineers" now.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#74

Post by dilyevsky »

jeffbee wrote: post #64
dilyevsky wrote: post #10

Earlier quoted context omitted.

Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere. > very broadly applicable. I don't see any "this would only apply at Google" in here. One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod

When I was there, the main IRC ran in prod. But it was intentionally a low-dependency system, an actual IRC server instead of something ridiculous like gIRC or IRC-over-stubby.

i think it had a corp dns label but i'm fuzzy on that. yes it could've been a prod instance which would mean you'd need to go to panic room in case corp was down but maybe that was the intention.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#75

Post by zdragnar »

derefr wrote: post #56
davedx wrote: post #47

Earlier quoted context omitted.

> “These days with devops the skill set needed for devs have indeed expanded to have significant overlap with SREs” Respectfully disagree on this. SRE is a huge complex realm unto itself. Just understanding how all the cloud components and environments and role systems work together is multiple training courses, let alone how to reliably deploy and run in them.

But modern approaches to dev require the SWEs to understand and model the operation of their software, and in fact program in terms of it — “writing infrastructure” rather than just code. Lambda functions, for example: you have to understand their performance and scalability characteristics — in turn requiring knowledge of things like the latency added by crossing the boundary between a managed shared service cluster…

That is barely tip-toeing across the very edges of SRE land.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#76

Post by goalieca »

xena wrote: post #53
londons_explore wrote: post #17

Earlier quoted context omitted.

Maybe I'm naive, but I would imagine that any raspberry pi could run an IRC server with 10,000 users... Surely 10,000 Users each on average receiving perhaps five 50 byte messages per second (that's a very busy chat room!) is a total bandwidth of 2.5 megabytes per second. And the CPU time to shuffle messages from one TCP connection to another, or encrypt/decrypt 2.5 megabytes per second should be small. There is no c…

IRC servers are single threaded. You have contention at that point.

Some of the bigger irc channels have thousand users and this was in the day of 128MB of ram.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#77

Post by oceanplexian »

VirusNewbie wrote: post #62
xorcist wrote: post #37

Earlier quoted context omitted.

Sys admins changed name to SREs which changed named to devops engineers or cloud engineers or whatever the title is now. Still the same competency. Someone needs to know how those protocols work, tell you latency characteristics of storage, and read those core dumps.

In my G SRE interview, I had to do the same rigorous Software Engineering algorithms rounds as well as show deep distributed systems knowledge in designing highly available systems.

If by rigorous algorithms you mean, spend a month memorizing a few dozen leetcode problems then sure, I’ll agree that is sadly the state of SRE interviews at FAANG.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#78

Post by oceanplexian »

gen220 wrote: post #29
fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

I imagine the threshold is something like 1 SRE for every $1mm of high-margin revenue you can link to guaranteeing the 2nd "9" of $product availability/reliability.

That’s sort of ridiculous. A mid-level SRE easily costs a quarter of that. And a company like Apple would then have 80,000 SREs? Lol no.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#79

Post by namtab00 »

vsareto wrote: post #49
davedx wrote: post #45

Earlier quoted context omitted.

Eh it’s not the same thing. (I’m very full stack with intermittent devops/sre experience). Full stack means you write code running on back end and front end. 99% of the time the code you write on the FE interfaces with your other code for the BE. It’s pretty coherent and feedback loops are similar. Devops/SRE on the other hand is very different and I agree we shouldn’t expect software developers be mixing in SRE in t…

If you can be good at front and back end and keep up with both of them simultaneously, that's great, but: - you spend more time to keep up with both of those sectors compared to dedicated front or back end positions - you context switch more often than dedicated positions - you spent more time getting good at both of those things - you removed some amount of communication overhead if there were two positions You are…

I vehemently refuse to do frontend...

Have I done it a lot before the SPA era? yes.

Would I be able to do a half decent job today? Probably.

Would that eat into what brainpower I currently muster to fulfill my backend role? I'm convinced of it.

Can I continue to earn a living in the current market? I'm afraid not for long...

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#80

Post by namtab00 »

crabbone wrote: post #43
fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

> much like [...] QA testers have mostly disappeared. Who told you that? QA isn't going anywhere... someone is doing testing, and that someone is a tester. They can be an s/w engineer by training, but as long as they are testing they are a tester. With sysadmins, there are fashion waves, where they keep being called different names like DevOps or SRE. I've not heard of such a thing with testing.

> someone is doing testing, and that someone is a -tester- user

excuse me for remembering something surely HN considers a platitude: "everyone has a TEST environment, few are fortunate enough to also have a PROD one"

Post reply on HN