51–60 of 86 posts
"Within Google, we implement periodic downtime in some services to prevent a service from being overly available." Uh..... what?
This is a great article for defining terms. For some reason though, this quote made me laugh out loud: "Excessive availability can become a problem because now it’s the expectation. Don’t make your system overly reliable if you don’t intend to commit to it to being that reliable."
I guess what meant here is that one should never make mistake of assuming that a highly reliable system can be built. As you start to approach near 100% reliable system, you start experiencing failures that are caused by minute disturbances/flaws in underlying dependencies(hardware, physical location) which can't be controlled. This is what they realized while trying to push the limits to build highly reliable system…
Wrong guess imho. It means building highly reliable systems requires knowledge and experiences. Trying to build them and solving the problems step-by-step is one way to understand how it can be achieved.
"Partial refund". That's a very low standard for a service level agreement, but typical of Google. Your whole business is down, it's their fault, and all you get a partial refund on the service.
A service level agreement is really a service packaged with an insurance product. The insurance product part should be evaluated as such - does it cover enough risk and is the coverage amount high enough? You can buy business interruption insurance from insurance companies, and should price that out in comparison with the cost and benefits of a SLA. If this is crucial to your core business, as with an entire retail chain going down because a cloud-based point of sale system goes down, it needs to be priced accordingly.
See: [1]
[1] https://www.researchgate.net/publication/226123605_Managing_...
This is an interesting article from a company that has almost nil customer support.
We buy a support package and receive excellent support for our GCP services.
Is it enough to use a gratis product? ("Paying" for it with data or ad-eyeballs, I suppose)
Is it enough to pay money for the product?
Must one also pay a subscription fee in addition to paying for the product in the first place? [1]
Is something else, sometimes, necessary (such as volume/clout)?
I think we've seen most of the spectrum of answers from the software industry (especially "enterprise" software), with the main novelty being the existence of web/SaaS gratis products.
[1] Depending on where on the spectrum between hand-holding and mere bug fixes the support ends up falling, this could be characterized as double-dipping
Google: "An SLA normally involves a promise to someone using your service that its availability should meet a certain level over a certain period, and if it fails to do so then some kind of penalty will be paid. This might be a partial refund of the service subscription fee paid by customers for that period, or additional subscription time added for free." "Partial refund". That's a very low standard for a service le…
It seems to be the standard. The most generous SLA I've seen is 5% off the monthly bill for each 30 minutes of downtime (up to 100%). If I'm down for 10 hours, waiving one month of bills doesn't come close to the damage done.
An SLA seems to be more of a promise than an agreement, because if the service goes down you're SOL and the provider gets a slap on the wrist (partial refund).
"Within Google, we implement periodic downtime in some services to prevent a service from being overly available." Uh..... what?
"Don’t make your system overly reliable if you don’t intend to commit to it to being that reliable"
If a service has exceeded the reliability target for a given time period, you can take it down to basically let users know that this can happen and to not expect more.
You don't want them to get to the point where they are integrating so much with a service (and assuming a higher reliability that you have not promised ) that they end up mad at you when it performs worse, but still as intended, at a later date.
Google: "An SLA normally involves a promise to someone using your service that its availability should meet a certain level over a certain period, and if it fails to do so then some kind of penalty will be paid. This might be a partial refund of the service subscription fee paid by customers for that period, or additional subscription time added for free." "Partial refund". That's a very low standard for a service le…
It's a standard across the industry, pretty much since the beginning of SLAs.
They're not insurance, and not meant to compensate you if your business is disrupted. That's on you. (And there are many ways to protect your business from provider outages.)
SLA payouts are meant to be mildly punitive, and to align incentives -- in aggregate, the SLA payouts add up and can hurt Google if there are a lot of customers affected by frequent outages.
When reading these articles, never forget that your company is NOT Google! If your company doesn't have a management/infrastructure/communication/skill structure that Google has, then it will be very difficult to implement these fundamentals. In many cases, an SRE is a job to save costs. If your company doesn't get its shit together and doesn't give your SREs the support it needs, then they'll hate their jobs and the…
I have no idea why you’re being downvoted. It’s the same thing as Borg/Kubernetes, MapReduce/Hadoop: some things just don’t apply or aren’t as effective unless you’re operating at a huge scale and with Google’s culture.
I'm not sure one has to go to the extreme of huge scale, anywhere near where Google is now, (not that that's what you said), nor all the aspects of their culture, but I agree that key fundamental aspects are often missed.
My favorite example is to point out that Google does not run Hadoop on expensive, virtualized AWS instances (or even brand-name servers with useless-for-purpose features[1] that creep up the cost). Rather, one of their competitive advantages, from the very start, has been to optimize hardware that they purchase, customize, and operate for cost (and performance).
The other is, as you mention, culture, which involves a remarkable amount of specialization, with groups dedicated to hardware, networking, internal tooling (i.e. building and maintaining the Hadoop-euquivalent), and, of course, SRE, who couldn't even begin to do their jobs without all those other groups' support.
Of course, there's an argument to be made that things like k8s and PaaS/IaaS can take the place of all those supporting groups at Google, but my counterargument is that they both fail to impart any benefit of customization (or, conversely cultural benefit of the mindset of doing everything that way across the entire company) and carry a tremendous cost (in money and complexity).
[1] redundant power supplies, high-density chasses, onboard hardware RAID