Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Twenty Years of Site Reliability Engineering

sre.google

Post reply on HN
Print view View original

21–30 of 128 posts

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#21

Post by kccqzy »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

SRE is not a byproduct of a bubble economy. I believe Google has had SREs since the very beginning. But still I think the rest of the point still stands. These days with devops the skill set needed for devs have indeed expanded to have significant overlap with SREs. I expect companies to downsize their SRE teams and distribute responsibilities to devs.

A second major reason is automation. If you read the linked site long enough you'll find that in the early days of Google, SREs did plenty of manual work like deployments while manually watching graphs. They were indispensable then simply because even Google didn't have enough automation in their systems! You can read the story of Sisyphus https://www.usenix.org/sites/default/files/conference/protec... to kind of understand how Google's initial failure of adopting standardized automation ensured job security for SREs.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#22

Post by vitalysh »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

I guess now they have a team of software engineers, where part is focused on infra and part on backend. Sys Admins disappeared? They are DevOps/IT Engineers now. QA? SWE in Test, and so on.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#23

Post by justapassenger »

tiddo_langerak wrote: post #11

I'm curious how people approach big red buttons and intentional graceful degradation in practice, and especially how to ensure that these work when the system is experiencing problems. E.g. do you use db-based "feature flags"? What do you do then if the DB itself is overloaded, or the API through which you access the DB? Or do you use static startup flags (e.g. env variables)? How do you ensure these get rolled out q…

Implantation details will depend on your stack, but 3 main things I’d keep in mind:

1. Keep it simple. No elaborate logic. No complex data stores. Just a simple checking of the flag.

2. Do it as close to the source as possible, but have limited trust in your clients - you may have old versions, things not propagating, bugs, etc. So best to have option to degrade both in the client and on the server. If you can do only one, so the server side.

3. Real world test! And test often. Don’t trust test environment. Test on real world traffic. Do periodic tests at small scale (like 0.1% of traffic) but also do more full scale tests on a schedule. If you didn’t test it, it won’t work when you need it. If it worked a year ago, it will likely not work now. If it’s not tested, it’ll likely cause more damage than it’ll solve.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#24

Post by vsareto »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

>Instead, many of those functions are performed by software development teams.

And they likely won’t be as good as dedicated SRE teams.

But few businesses care about that right now considering layoffs.

Throwing developers at a problem even if it isn’t in their skill set is an industry trend that won’t go away and be more pronounced during downturns.

Full stack developers are a great example of rolling two roles together without twice the pay.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#25

Post by sarchertech »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

I haven’t noticed that in my corner of one of those mentioned companies. Also I’m not an SRE, but during the height of the recent tech layoffs the only job postings I was seeing was for SRE.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#26

Post by tomcam »

shadowgovt wrote: post #13
francisofascii wrote: post #6

Earlier quoted context omitted.

Yeah, interesting tidbit. It might sound insane today that one engineer's desktop computer could cause such an outage. But that was probably more commonplace 20 years ago and even today in smaller orgs.

There was a famous incident at one point where code search internally went down. It turned out that while they had deployed the tool internally, one piece of the indexing process was still running as a cron job on the original developer's desktop machine. He went on vacation, his credentials aged out, and of crawler stopped refreshing the index. But my favorite incident will forever be the time they had to drill out…

> But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system

Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#27

Post by tomcam »

justapassenger wrote: post #23
tiddo_langerak wrote: post #11

I'm curious how people approach big red buttons and intentional graceful degradation in practice, and especially how to ensure that these work when the system is experiencing problems. E.g. do you use db-based "feature flags"? What do you do then if the DB itself is overloaded, or the API through which you access the DB? Or do you use static startup flags (e.g. env variables)? How do you ensure these get rolled out q…

Implantation details will depend on your stack, but 3 main things I’d keep in mind: 1. Keep it simple. No elaborate logic. No complex data stores. Just a simple checking of the flag. 2. Do it as close to the source as possible, but have limited trust in your clients - you may have old versions, things not propagating, bugs, etc. So best to have option to degrade both in the client and on the server. If you can do onl…

4. Document procedures and methodically test the docs as well

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#28

Post by Smaug123 »

For much much more on this, I'm most of the way through Google's book _Building Secure and Reliable Systems_, which is a proper textbook (not light reading). It's a pretty interesting book! A lot of what it says is just common sense, but as the saying goes, "common sense" is an oxymoron; it's felt useful to have refreshed my knowledge of the whole thing at once.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#29

Post by gen220 »

fensterblick wrote: post #19

Recently, I've heard of several companies folding up SRE and moving individuals to their SWE teams. Rumors are that LinkedIn, Adobe, and Robinhood have done this. This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams? I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Inst…

I imagine the threshold is something like 1 SRE for every $1mm of high-margin revenue you can link to guaranteeing the 2nd "9" of $product availability/reliability.

Re: Lessons Learned from Twenty Years of Site Reliability Engineering

#30

Post by benlivengood »

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz, etc.

It's currently handled by running many independent regions so that data centers can be brought up from fully dark by bootstrapping them from existing infra. I haven't heard of anyone bringing the stack up from a full power-off situation. Even when Facebook completely broke its production network a couple years ago the machines stayed on and running and had some internal connectivity.

This matters to everyone because while cloud resources are great at automatic restarts and fault recovery there's no guarantee that AWS, GCP, and friends would come back up after, e.g., a massive solar storm that knocks out the grid worldwide for long enough to run the generators down.

My guess is that there are some dedicated small DCs with exceptional backup power and the ability to be fully isolated from grid surges (flywheel transformers or similar).

Post reply on HN