Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ongoing Incident in Google Cloud

status.cloud.google.com

Post reply on HN
Print view View original

61–70 of 115 posts

Re: Ongoing Incident in Google Cloud

#61

Post by elsonrodriguez »

crazygringo wrote: post #56
monero-xmr wrote: post #31

This is why any criticism of AWS reliability is meaningless to me. All the cloud providers go down - all of them. Either you are multi-cloud, or you run your own hardware, but these events are inevitable.

> Either you are multi-cloud, or you run your own hardware If you run your own hardware these events are inevitable too.

I've seen skepticism about GCP and AWS availability from people with a single 2U in a closet somewhere.

I know it's just a psychological thing about giving up "control", but I have to stifle a chuckle every time.

Re: Ongoing Incident in Google Cloud

#62

Post by zamnos »

elsonrodriguez wrote: post #61
crazygringo wrote: post #56

Earlier quoted context omitted.

> Either you are multi-cloud, or you run your own hardware If you run your own hardware these events are inevitable too.

I've seen skepticism about GCP and AWS availability from people with a single 2U in a closet somewhere. I know it's just a psychological thing about giving up "control", but I have to stifle a chuckle every time.

Right? I can pay extra to have two ISPs for upstream connection, but I have no idea how I'd get a second, totally redundant power connection to the closet in my basement. A UPS with a battery's only going to last so long, so is generator fuel.

Re: Ongoing Incident in Google Cloud

#63

Post by throitallaway »

dilyevsky wrote: post #38
medler wrote: post #19

Earlier quoted context omitted.

> gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary. As I understand it, GCP is already designed to make global outages impossible. Obviously this outage shows that they messed up somehow and some global point of failure still remains. Looking forward to the post-mortem.

They had many many global outages through the years so that’s evidently not true. GCLB, iam, gcs and probably more Im missing just of the top of my head. Then there’s constant stream of regional networking borks where your latency is suddenly 5x which are not “global” but affect multiple regions

Anecdata, but in my experience Google Cloud has been MUCH more solid than my time spent on AWS.

Re: Ongoing Incident in Google Cloud

#64

Post by jrumbut »

elsonrodriguez wrote: post #61
crazygringo wrote: post #56

Earlier quoted context omitted.

> Either you are multi-cloud, or you run your own hardware If you run your own hardware these events are inevitable too.

I've seen skepticism about GCP and AWS availability from people with a single 2U in a closet somewhere. I know it's just a psychological thing about giving up "control", but I have to stifle a chuckle every time.

One aspect of that is the box in the closet is (in my experience anyway) either up or down. It fails more often, but it fails simpler.

In the cloud, even very small scale apps can run into weird situations like the app server is up, the database is down, and the cache is responding about 50% of the time.

If you don't account for that from the beginning, it can lead to your app displaying some bizarre stuff to users.

I haven't run a server locally in 13 years but I can see why some people would miss it.

Re: Ongoing Incident in Google Cloud

#65

Post by LewisVerstappen »

bongobingo1 wrote: post #47
zamnos wrote: post #40

Earlier quoted context omitted.

Google has follows-the-sun on-call rotations for large rotations, so this hit the UK team just after lunch.

Ah so the rotation rotates to match the current rotation. Very smart.

The sun never sets on the Google empire

Re: Ongoing Incident in Google Cloud

#66

Post by dehrmann »

vhiremath4 wrote: post #48
monero-xmr wrote: post #31

This is why any criticism of AWS reliability is meaningless to me. All the cloud providers go down - all of them. Either you are multi-cloud, or you run your own hardware, but these events are inevitable.

The amount of time you are down vs. up dictates your SLOs and SLAs. Criticism of how reliable one vs. another is is not only valid, it's backed by hundreds of millions of contractual dollars and credits every year. We spend tens of millions on AWS per year. We have several SLAs with them. Our Elasticache SLA was breached once (localized to us - not whole customer base) and we got credits which were commensurate with…

You definitely need to look at your SLA with your customers, but in my experience, multi-cloud isn't worth it. It's easier to be slightly less reliable, and throw your top-three cloud provider under the bus in the public post mortem. You'll probably cause bigger outages on your own in between provider outages, and multi-cloud adds another layer of complexity for things to go wrong.

Multi-cloud is saying you think you can manage Kafka across two or three clouds better than GCP can manage Pub/Sub.

Re: Ongoing Incident in Google Cloud

#67

Post by xwolfi »

tgtweak wrote: post #21
bushbaba wrote: post #8

This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.

You can't really have 30+ fully independent regions running their own stack with different versions of apps and separate secrets, IP/routing and certificates in each. At some point you have to unify or it becomes either unmanageable or inconsistent.

But you can have 3. Why did you choose 30?

In my company we are split in 3, US, EU, APAC, and we have the same issue with global outage for stuff we could have just managed regionally. For all the savings of the global architecture, they disappear each minute a client is down on a global outage because a guy thousands of kms away messed up.

You dont have to unify, at all. You dont unify with your competitors, and the world has not exploded: compete internally between regions ?

Re: Ongoing Incident in Google Cloud

#68

Post by aflag »

elsonrodriguez wrote: post #61
crazygringo wrote: post #56

Earlier quoted context omitted.

> Either you are multi-cloud, or you run your own hardware If you run your own hardware these events are inevitable too.

I've seen skepticism about GCP and AWS availability from people with a single 2U in a closet somewhere. I know it's just a psychological thing about giving up "control", but I have to stifle a chuckle every time.

I've worked in companies that had everything on prem and cloud companies. There are many nice things about cloud, but reliability is not one of them. Everything is a lot simpler on prem and fails a lot less in my experience. The downside being that scaling is harder. And it can be more expensive, depending on your size.

Re: Ongoing Incident in Google Cloud

#69

Post by AlecSchueler »

throitallaway wrote: post #63
dilyevsky wrote: post #38

Earlier quoted context omitted.

They had many many global outages through the years so that’s evidently not true. GCLB, iam, gcs and probably more Im missing just of the top of my head. Then there’s constant stream of regional networking borks where your latency is suddenly 5x which are not “global” but affect multiple regions

Anecdata, but in my experience Google Cloud has been MUCH more solid than my time spent on AWS.

While a fair point it's in no way a counter argument to what the person above was saying. Having fewer outages is not the same as having no global outages.

Re: Ongoing Incident in Google Cloud

#70

Post by Leszek »

bongobingo1 wrote: post #47
zamnos wrote: post #40

Earlier quoted context omitted.

Google has follows-the-sun on-call rotations for large rotations, so this hit the UK team just after lunch.

Ah so the rotation rotates to match the current rotation. Very smart.

I like the mental image of this being a very precise matching -- as the sun traces across the sky, the responsibility of on-call passes from desk to desk, town to town, country to country; two engineers on a boat in the Atlantic race to keep up with their rotation...
Post reply on HN