[deleted]
SRE Fundamentals: SLIs, SLAs and SLOs
71–80 of 86 posts
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#72Earlier quoted context omitted.
The Google SREs mentioned this in their book; the Chubby locking service had uptime that was so high that folks started to neglect making their own services resilient to Chubby failures: https://landing.google.com/sre/book/chapters/service-level-o...
+1 for this book. As a junior DevOps engineer this book has been super helpful.
Mine just came in the mail today. Pretty stoked.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#73Google: "An SLA normally involves a promise to someone using your service that its availability should meet a certain level over a certain period, and if it fails to do so then some kind of penalty will be paid. This might be a partial refund of the service subscription fee paid by customers for that period, or additional subscription time added for free." "Partial refund". That's a very low standard for a service le…
> "Partial refund". That's a very low standard for a service level agreement, but typical of Google. Your whole business is down, it's their fault, and all you get a partial refund on the service.
The SLA is the contract. While this may not be possible, you'd normally have to negotiate a higher payout for a higher service cost, but otherwise it's fixed based on the amount you pay, not on the amount your business makes.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#74Earlier quoted context omitted.
The Google SREs mentioned this in their book; the Chubby locking service had uptime that was so high that folks started to neglect making their own services resilient to Chubby failures: https://landing.google.com/sre/book/chapters/service-level-o...
Still that's bad design on the clients' part. E.g. - Just because malloc "never" fails doesn't mean it can't fail :) so better error check for it.
Failure of malloc() might be a bad example to pick because on linux, by default, most distros overcommit, so malloc won't fail, generally. Instead, malloc will succeed allocating the address space just fine, but the RAM will get allocated upon first use, meaning that even though malloc gave you a supposedly valid pointer rather than NULL, actually using that pointer will crash your program.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#75"Within Google, we implement periodic downtime in some services to prevent a service from being overly available." Uh..... what?
If you exceed your SLO, people think your service is more reliable than it is.
When you have your infrequent but expected failures, they are caught by surprise unless you normalize your SLO burn.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#76Earlier quoted context omitted.
Still that's bad design on the clients' part. E.g. - Just because malloc "never" fails doesn't mean it can't fail :) so better error check for it.
Failure of malloc() might be a bad example to pick because on linux, by default, most distros overcommit, so malloc won't fail, generally. Instead, malloc will succeed allocating the address space just fine, but the RAM will get allocated upon first use, meaning that even though malloc gave you a supposedly valid pointer rather than NULL, actually using that pointer will crash your program.
Is there a way to fix this/switch it off? I never got the rationale for this behaviour.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#77Google: "An SLA normally involves a promise to someone using your service that its availability should meet a certain level over a certain period, and if it fails to do so then some kind of penalty will be paid. This might be a partial refund of the service subscription fee paid by customers for that period, or additional subscription time added for free." "Partial refund". That's a very low standard for a service le…
> "Partial refund". That's a very low standard for a service level agreement, but typical of Google. It seems to be the standard. The most generous SLA I've seen is 5% off the monthly bill for each 30 minutes of downtime (up to 100%). If I'm down for 10 hours, waiving one month of bills doesn't come close to the damage done. An SLA seems to be more of a promise than an agreement, because if the service goes down you'…
https://www.upcloud.com/documentation/terms/ - SLA , 50x, limits apply
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#78Earlier quoted context omitted.
Failure of malloc() might be a bad example to pick because on linux, by default, most distros overcommit, so malloc won't fail, generally. Instead, malloc will succeed allocating the address space just fine, but the RAM will get allocated upon first use, meaning that even though malloc gave you a supposedly valid pointer rather than NULL, actually using that pointer will crash your program.
Is there a way to fix this/switch it off? I never got the rationale for this behaviour.
There's a sysctl: vm.overcommit_memory=2
What most people don't realize is that you will get more OOMs if you disable overcommit.
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#79Google: "An SLA normally involves a promise to someone using your service that its availability should meet a certain level over a certain period, and if it fails to do so then some kind of penalty will be paid. This might be a partial refund of the service subscription fee paid by customers for that period, or additional subscription time added for free." "Partial refund". That's a very low standard for a service le…
>A service level agreement is really a service packaged with an insurance product.
Now, such a service may be sold on the very high end... but in the general case, that's not what "Service Level Agreement" usually means.
(as an aside, I strongly suggest you get your business insurance from a party other than your service provider; serous outages can bankrupt service providers as-is... if they had to pay out customer damages, that would become a lot more likely.)
Re: SRE Fundamentals: SLIs, SLAs and SLOs
#80So there is one obscure metric "service is available, i.e. can do its job", and this metric has different attributes: there are actual metric values (SLIs), there are internal goals (SLOs) and there are legally binding promises (SLAs) to users/customers. I would argue that this is not much content here.
Content, imo, would be something like this: We define "available" as "processor_loadThat you should track metrics, that you should set goals, and that you should define SLAs with your customers/users is standard business practice, not new knowledge.