Live data from Hacker News

Ask HN: GCP Outage?

news.ycombinator.com

31–40 of 44 posts

Re: Ask HN: GCP Outage?

#31
post #23
post #19

Maybe centralising all our IT infrastructure wasn't a good idea after all.

I dunno. If just your employers site is down then you'll be expected to fix it, whereas if everyone is down there's less pressure.

yup. I figure I'm basically a free-rider (except I am paying a relatively small amount.)

Re: Ask HN: GCP Outage?

#32
post #25

Earlier quoted context omitted.

I forgot all about pager duty... been retired over a year now. I don't miss pager duty.

> I forgot all about pager duty Probably because it's hard to form long-term memories when you're sleep-deprived :/

or 8 shots in

Re: Ask HN: GCP Outage?

#33
post #23
post #19

Maybe centralising all our IT infrastructure wasn't a good idea after all.

I dunno. If just your employers site is down then you'll be expected to fix it, whereas if everyone is down there's less pressure.

Nobody who talks to actual stakeholders can use this as a defence.

B2B customers don’t care if the other sites are also down, your SLA is affected with them, and they will want compensation.

Re: Ask HN: GCP Outage?

#35
Yes. Many times. Kubernetes upgrade during maintenance schedule borks up entire cluster, yet everything is green on status page. Support case under enterprise support plan took almost 6 hours to get it resolved.

Re: Ask HN: GCP Outage?

#36
post #33
post #23

Earlier quoted context omitted.

I dunno. If just your employers site is down then you'll be expected to fix it, whereas if everyone is down there's less pressure.

Nobody who talks to actual stakeholders can use this as a defence. B2B customers don’t care if the other sites are also down, your SLA is affected with them, and they will want compensation.

You need to phrase it as Internet Weather.

Re: Ask HN: GCP Outage?

#37
post #35

Yes. Many times. Kubernetes upgrade during maintenance schedule borks up entire cluster, yet everything is green on status page. Support case under enterprise support plan took almost 6 hours to get it resolved.

I see they made great strides in the past 5 years - it used to take days =)

Re: Ask HN: GCP Outage?

#38
post #20

Earlier quoted context omitted.

This isn’t a whole-cloud outage. It’s not even a whole-region outage. Whole-cloud outages are pretty damn rare. The recent GCP issues are an exception to the general rule. I’d posit that the complexity of a multi-cloud setup is generally going to reduce your service’s reliability more than relying on a single cloud does.

Whole-zone outages are also rare...

"Rarity" is a distinction without merit in this particular case; the important thing to note is that (most) clouds don't guarantee _any_ availability of a single zone. A system which stashes all of its infrastructure in one zone only is expected to be impacted by issues with that cloud, while a multi-zone setup spanning a region is generally "soft-guaranteed" to be resilient to normal operations / failures.

Re: Ask HN: GCP Outage?

#39
post #20

Earlier quoted context omitted.

Whole-zone outages are also rare...

"Rarity" is a distinction without merit in this particular case; the important thing to note is that (most) clouds don't guarantee _any_ availability of a single zone. A system which stashes all of its infrastructure in one zone only is expected to be impacted by issues with that cloud, while a multi-zone setup spanning a region is generally "soft-guaranteed" to be resilient to normal operations / failures.

> (most) clouds don't guarantee _any_ availability of a single zone

Really?

AWS (EC2) does: https://aws.amazon.com/compute/sla/?did=sla_card&trk=sla_car... so does GCP (GCE): https://cloud.google.com/compute/sla?hl=en and so does OVH: https://us.ovhcloud.com/legal/sla/public-cloud/

Are none of those three part of "most clouds"? What cloud platform do you use?

Re: Ask HN: GCP Outage?

#40

Reminder that multi cloud >>> multi region Anyone who says otherwise is selling availability theater Too many whole-cloud outages due to a bad config in the last 2 months (GCP x2, cloudflare x2)

I worked at PagerDuty, so definitely not selling availability theater. We did multi-cloud / multi-region for many years, and the story is not so simple. Development is all about trade-offs, and deciding what risk you are OK with. Multi-cloud provided a relatively small amount of value (given how incredibly unlikely whole-cloud outages are, even full-region outages are quite rare) at the expense of 2x implementation overhead, 2x exposure to random cloud-specific operational events, and the need to develop for the common denominator of functionality, which leaves out a LOT of interesting cloud offerings. In the end, it ended up just not being worth it, and moving to the single-cloud multi-region config provided enough reliability even for the company where reliability is the primary differentiator.

In my current job as a technical due diligence advisor, I frequently recommend multi-AZ setup but specifically not multi-region, because the former is easy and worthwhile while the latter carries a lot more operational overhead (you become much more sensitive to various latencies and network jitters) and you now need to think about things like synchronous vs async replication, etc. Much better to focus dev effort on the product, rather than eke out an additional .001% of availability (unless availability is a super critical component).

Post reply on HN