Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

301–310 of 341 posts

Re: Google Cloud networking issues in us-east1

#301

Post by samcday »

orf wrote: post #194
samcday wrote: post #147

Earlier quoted context omitted.

Things like: * filtering traces by services has been broken in App Engine flex environments for more than a year. * copy/pasting identifiers between places is a nightmare * their IAM design is somehow worse than AWS. It’s so impressively bad I can’t even be mad. My favourite part of their IAM approach is how they have consolidated a majority of the IAM controls in the IAM page, but then random services like GCS have…

> * multi-account paper cuts I always considered the Google Cloud approach of a "single account, multiple projects" a lot cleaner than the AWS "hundreds of accounts" approach. Do you not find this the case?

Oh the multi-project stuff is definitely nice. I'm referring to the ability to keep multiple distinct Google accounts logged in.

Re: Google Cloud networking issues in us-east1

#302

Post by kowdermeister »

mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

It's just global warming again. The weather in the clouds gets increasingly unpredictable :)

Re: Google Cloud networking issues in us-east1

#303

Post by mrweasel »

boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad.

As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.

Re: Google Cloud networking issues in us-east1

#304

Post by haecceity »

boulos wrote: post #202
ricardobeat wrote: post #193

Earlier quoted context omitted.

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

As kyrra says below, you're in the clear if you state that this is just your opinion. Naturally, prefacing something terrible as "just your opinion" doesn't make it fine. In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opin…

They only found you because your HN username is same as your Google alias?

Re: Google Cloud networking issues in us-east1

#305

Post by not_kurt_godel »

cthalupa wrote: post #207
mathattack wrote: post #91

Earlier quoted context omitted.

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

If you're running in multiple clouds for HA/DR reasons, you are limited to the lowest common denominator of features/services between them. Or maintaining multiple codebases/architectures, and the massive pile of issues that entails. I am not a fan of multi-cloud for this reason. Multiple regions, as long as your provider offers all of the services, you can have a carbon copy. Much easier. It depends on your needs, y…

> you can have a carbon copy. Much easier.

Certain terms and conditions may apply :) Carbon copy of a static website or one whose data is only a one-way flow from some off-cloud source of truth? Sure! Multi-master or primary-secondary with failover? Stray too far from the narrow path of specialized managed solutions and things get very complex, very quickly. That being said - it's mostly just the nature of the beast. If you're not able to tolerate a regional outage, multi-region is a pill you're going to have to swallow, no buts about it.

Re: Google Cloud networking issues in us-east1

#306

Post by robbiep »

haecceity wrote: post #304
boulos wrote: post #202

Earlier quoted context omitted.

As kyrra says below, you're in the clear if you state that this is just your opinion. Naturally, prefacing something terrible as "just your opinion" doesn't make it fine. In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opin…

They only found you because your HN username is same as your Google alias?

woah that's weird. This (Hacceity) is a social media alias of mine. For a moment I thought I wrote this. Did you come across the word in the Mars trilogy too?

Re: Google Cloud networking issues in us-east1

#307

Post by hajhatten »

harshreality wrote: post #143

Hacker News: The real status page and help desk for the internet. Do companies realize how absurd this is? ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.

This reminds me of an incident in Sweden a couple of years ago.

We test our disaster alarms on a known schedule. And just a couple of years ago, during the peak vacation time in the summer, the alarm went off, off schedule.

This made the entire country panic. Were we being attacked? The agency that is supposed to let people know through the public channels like tv, radio etc were silent. They were themselves on vacation probably. The websites and apps they've setup were ridiculously underpowered and were basically DDOS'ed by the spike in traffic they were getting.

News outlets were also struggling, but did way better.

The only thing that withstood the sudden burst in traffic without a hitch was facebook and twitter.

The official statement i think was that the alarm was triggered by accident (never happened before, i think). But goes to show how badly our emergency response is setup.

Re: Google Cloud networking issues in us-east1

#308

Post by jodrellblank »

mrweasel wrote: post #303
boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.

Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.

Re: Google Cloud networking issues in us-east1

#309

Post by rjf72 »

m0zg wrote: post #221
Operyl wrote: post #218

Earlier quoted context omitted.

My horribly out of date system works, therefore I should never strive to improve it or god forbid update it (since that involves “fucking with it” in ways that can break it from version to version)? That gets you technical debt and that’s not fun.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…

COBOL is a big example of this. It's still ubiquitous in many industries where reliability is near the top of priorities. And I imagine in 50 years we'll still very likely still have many critical systems operating on COBOL. I wonder how many will be running 'The Next Big Thing' language/api then...?

Re: Google Cloud networking issues in us-east1

#310

Post by gsich »

jodrellblank wrote: post #308
mrweasel wrote: post #303

Earlier quoted context omitted.

>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.

Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.

Disagree.
Post reply on HN