Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Twenty Years of Site Reliability Engineering

sre.google

Post reply on HN
Print view View original

41–50 of 128 posts

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#41

Post by tayo42 »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

Two part to this,

Is sre a bubble thing.

I never got why SRE existed.(SRE has been my title...) The job responsibilities, care about monitoring, logging, performance, metrics of applications are all things a qualified developer should be doing. Offloading caring about operating the software someone writes to someone else just seems illogical to me. Put the swes on call. If swes think the best way to do something is manual, have them do it them selves, then fire them for being terrible engineers. All these tedious interviews and a SWE doesn't know how the computer they are programing works? Its insane. All that schooling and things like how does the OS work, which is part of an undergrad curriculum, gets offloaded to a career and title mostly made up of self taught sysadmin people? Every good swe Ive known, knew how the os, computer, network works.

> if SRE will be around 10 years from now,

Other tasks that SRE typically does now, generalized automation, provide dev tools and improve dev experience, is being moved to "platform" and teams with those names. I expect it to change significantly.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#42

Post by shadowgovt »

tomcam wrote: post #26
shadowgovt wrote: post #13

Earlier quoted context omitted.

There was a famous incident at one point where code search internally went down. It turned out that while they had deployed the tool internally, one piece of the indexing process was still running as a cron job on the original developer's desktop machine. He went on vacation, his credentials aged out, and of crawler stopped refreshing the index. But my favorite incident will forever be the time they had to drill out…

> But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.

Well, mostly-kinda-sorta. ;) It's the internal password vault, and there's only one of them, so it's more like "they broke it on purpose and then had to fix it before the company went off the rails." Among the things kept in that vault are credentials that if they age out or aren't regularly refreshed, key internal infrastructure starts grinding to a halt.

But still, "it broke while engineers were staring at it and trying to break it" is a better scenario than "it broke surprisingly while engineers were trying to do something else."

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#43

Post by crabbone »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

> much like [...] QA testers have mostly disappeared.

Who told you that?

QA isn't going anywhere... someone is doing testing, and that someone is a tester. They can be an s/w engineer by training, but as long as they are testing they are a tester.

With sysadmins, there are fashion waves, where they keep being called different names like DevOps or SRE. I've not heard of such a thing with testing.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#44

Post by Moto7451 »

dilyevsky wrote: post #40
DaiPlusPlus wrote: post #36

Earlier quoted context omitted.

> Afair Google just ran irc on their corp network which was completely separate from prod I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?

I don't think zero-trust prohibits network segmentation for redundancy or due to geographical constraints etc. It's mainly about how you gain access.

Correct. At an old job we did zero trust corp on a different AWS region and account. The admin site was a different zero trust zone in prod region/account and was supposed to eventually become another AWS account in another region (for cost purposes).

I can’t say if any of this was ideal but it did work unobtrusively.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#45

Post by davedx »

vsareto wrote: post #24
fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

>Instead, many of those functions are performed by software development teams. And they likely won’t be as good as dedicated SRE teams. But few businesses care about that right now considering layoffs. Throwing developers at a problem even if it isn’t in their skill set is an industry trend that won’t go away and be more pronounced during downturns. Full stack developers are a great example of rolling two roles toget…

Eh it’s not the same thing. (I’m very full stack with intermittent devops/sre experience).

Full stack means you write code running on back end and front end. 99% of the time the code you write on the FE interfaces with your other code for the BE. It’s pretty coherent and feedback loops are similar.

Devops/SRE on the other hand is very different and I agree we shouldn’t expect software developers be mixing in SRE in their day to day. The skills, tools, mindset, feedback loop, and stress levels are too different.

If you’re not doing simple monoliths then you need a dedicated devops/SRE team.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#46

Post by nrr »

tayo42 wrote: post #41
fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

Two part to this, Is sre a bubble thing. I never got why SRE existed.(SRE has been my title...) The job responsibilities, care about monitoring, logging, performance, metrics of applications are all things a qualified developer should be doing. Offloading caring about operating the software someone writes to someone else just seems illogical to me. Put the swes on call. If swes think the best way to do something is m…

Oddly, the call to put the SWEs in the on-call rotation was one of the original goals of site reliability engineering as an institutional discipline. The idea at conception was that SREs were expensive, and only after product teams got their act together could they truly justify the cost of full-time reliability engineering support.

It's only in the past 10 years (reasonable people may disagree on that figure) that being a site reliability engineer came to mean being something other than a professional cranky jackass.

What I care about as an SRE is not graphs or performance or even whether my pager stays silent (though, that would be nice). No, I want the product teams to have good enough tools (and, crucially, the knowledge behind them) to keep hitting their goals.

Sometimes, frankly, the monitoring and performance get in the way of that.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#47

Post by davedx »

kccqzy wrote: post #21
fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

SRE is not a byproduct of a bubble economy. I believe Google has had SREs since the very beginning. But still I think the rest of the point still stands. These days with devops the skill set needed for devs have indeed expanded to have significant overlap with SREs. I expect companies to downsize their SRE teams and distribute responsibilities to devs. A second major reason is automation. If you read the linked site…

> “These days with devops the skill set needed for devs have indeed expanded to have significant overlap with SREs”

Respectfully disagree on this. SRE is a huge complex realm unto itself. Just understanding how all the cloud components and environments and role systems work together is multiple training courses, let alone how to reliably deploy and run in them.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#48

Post by throwawaaarrgh »

The cheapest way to prevent an outage is to catch it early in its lifecycle. Software bugs are like real bugs. First is the egg, that's the idea of the change. Then there's the nymph, when it hatches; first POC. By the time it hits production, it's an adult.

Wait - isn't there a stage before adulthood? Yes! Your app should have several stages of maturity before it reaches adulthood. It's far cheaper to find that bug before it becomes fully grown (and starts laying its own eggs!)

If you can't do canaries and rollbacks are problematic, add more testing before the production deploy. Linters, unit tests, end to end tests, profilers, synthetic monitors, read-only copies of production, performance tests, etc. Use as many ways as you can to find the bug early.

Feature flags, backwards compatibility, and other methods are also useful. But nothing beats Shift Left.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#49

Post by vsareto »

davedx wrote: post #45
vsareto wrote: post #24

Earlier quoted context omitted.

>Instead, many of those functions are performed by software development teams. And they likely won’t be as good as dedicated SRE teams. But few businesses care about that right now considering layoffs. Throwing developers at a problem even if it isn’t in their skill set is an industry trend that won’t go away and be more pronounced during downturns. Full stack developers are a great example of rolling two roles toget…

Eh it’s not the same thing. (I’m very full stack with intermittent devops/sre experience). Full stack means you write code running on back end and front end. 99% of the time the code you write on the FE interfaces with your other code for the BE. It’s pretty coherent and feedback loops are similar. Devops/SRE on the other hand is very different and I agree we shouldn’t expect software developers be mixing in SRE in t…

If you can be good at front and back end and keep up with both of them simultaneously, that's great, but:

- you spend more time to keep up with both of those sectors compared to dedicated front or back end positions

- you context switch more often than dedicated positions

- you spent more time getting good at both of those things

- you removed some amount of communication overhead if there were two positions

You are definitely not being compensated for that extra work and benefit to the business given that full stack salaries are close to front end and back end position salaries.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#50

Post by anonacct37 »

tiddo_langerak wrote: post #11

I'm curious how people approach big red buttons and intentional graceful degradation in practice, and especially how to ensure that these work when the system is experiencing problems. E.g. do you use db-based "feature flags"? What do you do then if the DB itself is overloaded, or the API through which you access the DB? Or do you use static startup flags (e.g. env variables)? How do you ensure these get rolled out q…

I worked at enough companies to see that many of them have some notion of "centralized config that's rolled to the edge and can be updated at runtime".

I've done this with djb's CDB (constant database). But I've seen people poll an API for JSON config files or dbm/gdbm/Berkeleydb/leveldb.

This can extend to other big red buttons. It's not that elegant but I've had numerous services that checked for the presence of a file to serve health checks. So pulling a node out of load balancer rotation was as easy as creating a file.

The idea is that then when there's a datab outage the system defaults to serving the last known good config.

Post reply on HN