Live data from Hacker News

Google Cloud Is Down

news.ycombinator.com

371–380 of 630 posts

Re: Google Cloud Is Down

#371

When talking about GCE being down please also mention what regions you are talking about

us-central1 us-west1 us-west2 is what I’ve heard so far. east seems to be OK, and Europe too

South America is down for me. But youtube works OK, I can watch Google IO and they talking about all Google greatness while their outage impact second class clients on their cloud.

Re: Google Cloud Is Down

#372
Looks like my Gmail is back, but I don't have any emails from while it was down. Yikes.

Edit: just got one email from the downtime, so perhaps my initial conclusion was incorrect

Re: Google Cloud Is Down

#373
post #322

Now is a good time to point out that the SLA of Google Cloud Storage only covers HTTP 500 errors: https://cloud.google.com/storage/sla . So if the servers are not responding at all then it's not covered by the SLA. I've brought this to their attention and they basically responded that their network is never down.

Ironically I can't read that page because, since it's Google-hosted, I'm getting an HTTP 500 error... but which means at least that service is SLA-covered...

Cloud services live and die by their reputation, so I'd be shocked if Google ever tried to get out of following an SLA contract based on a technicality like that. It would be business suicide, so it doesn't seem like something to be too worried about?

Re: Google Cloud Is Down

#374

Earlier quoted context omitted.

I have an instance in us-west-1 (Oregon) which is up, but an instance in us-west-2 (Los Angeles) which is down. Not sure if that means Oregon is unaffected though.

us-west-1 is Northern California (Bay area). us-west-2 is Oregon (Boardman).

Incorrect. GCE us-west1 is the Dalles, Oregon and us-west2 is Los Angeles.

Re: Google Cloud Is Down

#375
post #243

Earlier quoted context omitted.

I use both services heavily at work. The networking in GCP is terrible. We experience minor service degradation multiple times a month due to networking issues in GCP (elevated latency, errors talking to the DB, etc). We've even had cases where there was packet corruption at the bare metal layer, so we ended up storing a bunch of garbage data in our caches / databases. Also, the networking is less understandable on G…

> We've even had cases where there was packet corruption at the bare metal layer, I'd love to know how this happens in the modern world. I've seen it myself only once (not GCP, but our own network with cisco equipment.) Is something in the chain not checking the packet's CRC?

Had something similar last year because of a core router fabric issue. A few years ago, there was a batch of new servers with buggy motherboards corrupting/dropping packets, can't begin to imagine how hard it was to diagnose.

That's in own datacenters, not cloud.

Re: Google Cloud Is Down

#377

Earlier quoted context omitted.

And the illusion of superiority over non cloud offerings.

I keep trying to explain to people that our customers don’t care that there is someone to blame they just want their shit to work. There are advantages to having autonomy when things break. There’s a fine line or at least some subtlety here though. This leads to some interesting conversations when people notice how hard I push back against NIH. You don’t have to be the author to understand and be able to fiddle with…

Of course customers don't care about the root cause. The point of the cloud isn't to have a convenient scapegoat to punt blame to when your business is affected. It's a calculated risk that uptime will be superior compared to running and maintaining your own infrastructure, thus allowing your business to offer an overall better customer experience. Even when big outages like this one are taken into account, it's often a pretty good bet to take.

Re: Google Cloud Is Down

#378
post #151

Earlier quoted context omitted.

To clarify something: this outage doesn’t appear to be global, but it is hitting us particularly hard in parts of the US. So for the folks with working VMs in Mumbai, you’re not crazy. But for everyone with sadness in us-central1, the team is on it.

Some services are still impacted globally. Gmail over IMAP is unreachable for me. (Edit: gmail web is fine)

+1- imap gmail is down for me in Australia

Re: Google Cloud Is Down

#379
post #13

Earlier quoted context omitted.

Gosh, this was so scary... I thought someone had hacked in and deleted everything... I hope they come back. This is still pretty scary

[I am the Cloud SQL tech lead] This is a networking issue, and your data is safe. Cloud SQL stores instance metadata regionally, so it shares a failure domain with the data it describes. When the region is down or inaccessible, instances are missing from the list results, but that doesn't say anything about the instance availability from within region.

That's good to know. What confuses me is why they're saying "We continue to experience high levels of network congestion in the eastern USA", when I'm in us-west2 (Los Angeles) and none of my CloudSQL instances, nor is my k8s cluster showing up or contactable...

Re: Google Cloud Is Down

#380
post #370
post #344

Earlier quoted context omitted.

> For the downvoters, please just link here the proof if you disagree. https://wasabi.com/

How can they possibly guarantee eleven nines? Considering I’ve never heard of this company and they offer such crazy-sounding improvements over the big three, it feels like there should be a catch.

For data durability? I believe some AWS offerings also have an SLA of eleven 9's of data durability.
Post reply on HN