"Within Google, we implement periodic downtime in some services to prevent a service from being overly available."
Uh..... what?
SRE Fundamentals: SLIs, SLAs and SLOs
41–50 of 86 posts
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#42"Within Google, we implement periodic downtime in some services to prevent a service from being overly available." Uh..... what?
Services have different relationships with each others in terms of dependencies, and in terms of what you think those dependencies are.
If your idea of how things work is that services A, B, and C can optionally use service D, else use some fallback process, then if D has never failed, then you've never used that fallback process. And services X, Y, and Z which rely on services A, B, and C haven't had to deal with those services using their fallback processes either. So, instead of waiting for D to fail, you can take it down at a convenient time.
This applies to services as a whole, or services within a locality, or all services in some availability zone.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#43Earlier quoted context omitted.
The Google SREs mentioned this in their book; the Chubby locking service had uptime that was so high that folks started to neglect making their own services resilient to Chubby failures: https://landing.google.com/sre/book/chapters/service-level-o...
+1 for this book. As a junior DevOps engineer this book has been super helpful.
the book is structured in a way that makes it pretty easy to jump around and pick and choose which parts you want to read or skip, so it's not a very large commitment to read it
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#44Earlier quoted context omitted.
I've also seen it as Harvard and MIT graduates, then someone comes in, washes his hands first, saying "at Yale, they taught us to wash our hands before touching a holy object."
Quite OT but I almost always wash my hands _before_ (and after) using the restroom. Especially in a public place, it always made sense to me to do it before and after. It seems much more hygienic both for the "holy object" and other people!
I was told that if your work in a chemical plant or a chip fab you learn to wash your hands before you don't want chemicals on sensitive parts
And of course you don't know what germs etc are on the taps :-)
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#45This is an interesting article from a company that has almost nil customer support.
I suspect they meant there internal customers :-)
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#46"Within Google, we implement periodic downtime in some services to prevent a service from being overly available." Uh..... what?
Similar concept to Netflix Chaos Monkey.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#47When reading these articles, never forget that your company is NOT Google! If your company doesn't have a management/infrastructure/communication/skill structure that Google has, then it will be very difficult to implement these fundamentals. In many cases, an SRE is a job to save costs. If your company doesn't get its shit together and doesn't give your SREs the support it needs, then they'll hate their jobs and the…
> In many cases, an SRE is a job to save costs.
This is 100% the case. I would actually argue that it is the only job of the SRE organization - hit the budgets by balancing costs of availability vs. costs of unavailability. If the org has massive budgets and general budget flexibility then it is easy. Otherwise, SREs are magicians to pull the rabbits out of a hat inventing the most awe inspiring methods/tools/hacks/workarounds needed to meet and beat budget targets.
In the other orgs SREs are an indirect level of outsourcing of everything to SaaS providers.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#48"Within Google, we implement periodic downtime in some services to prevent a service from being overly available." Uh..... what?
Imagine if in python open('file.txt', 'r') never failed so no one ever bothered to put a try block. To prevent this from happening they purposely have open() fail a couple times.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#49If you're building a system from scratch, keep in mind that this way of designing your service may not be flexible enough. You don't want just service level objectives, agreements and indicators, you want customer level.
Your service may end up providing for multiple customers with different requirements. Maybe 1% of your customers will end up using 99% of your resources, creating uncomfortable situations that affect the other 99% of customers. To get away from this you have to start spinning off multiple identical services just for groups of customers, which is really annoying to maintain. You may find you need to add hard resource limits to control customer behavior, which is hard to add after the fact.
Instead, if you design your new system from scratch with customer-specific isolation and service levels, you can run one giant service and still prevent customer-specific load from hampering the rest of the service. You can also just run duplicate services at different levels of availability based on customer requirements, but that's not going to work forever.
As an aside, I'm looking forward to reading ITIL 2019 to see what new processes they've adopted. I think everyone who's getting into SRE stuff should have a solid foundation on the basics of IT Operations management first.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#50These distinctions started making more sense when I realize they map to OKRs which is generally how Google is said to track individual and team performance. In general, it's good to be precise about how you measure and when something is a hard or soft boundary. Otherwise, firefighting gets out of control. It's hard to determine when to stop something and put out a fire if you can't prioritize issues based on the boun…
SLOs certainly don't rigidly map to OKRs. Maybe it's easier to consider them (two sided) commitments about the quality of service? They're more of an ongoing measure of quality rather than a quarterly objective.
Correct. Breaking or risking the SLO will instead lead to stopping new features until reliability is restored.