Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

251–260 of 341 posts

Re: Google Cloud networking issues in us-east1

#251

Post by akhilcacharya »

user5994461 wrote: post #219
ricardobeat wrote: post #193

Earlier quoted context omitted.

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

Google employees are commenting publicly and on Hacker News all the time. If there is a policy of not speaking publicly about the company, this has been the most blatantly ignored policy ever.

I’m 90% sure it’s just to flex, honestly.

I work at another FANG with a roughly equal engineering community and I don’t see my kind commenting as much at all!

Re: Google Cloud networking issues in us-east1

#252

Post by ajhurliman »

mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

Seems like if it continues to be a problem that more multi-cloud solutions will present themselves (Terraform does that sort of thing, right?).

Re: Google Cloud networking issues in us-east1

#253

Post by sudosteph »

david-cako wrote: post #164
notatoad wrote: post #149

Earlier quoted context omitted.

seriously, they've got a text field on the official status page, why not put the text boulos posted here in that instead of the meaningless text they've got there?

I work for AWS. There is typically a balance that has to be struck when sharing information with customers. I would imagine this goes for most companies, which is why it isn't until a post-mortem that the messaging is fully refined.

True, but I'd argue that the "Customer Obsession" priciple would drive you to attempt some sort of good-faith effort towards real-time communication.

Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?

Re: Google Cloud networking issues in us-east1

#254

Post by [deleted] »

tscanausa wrote: post #87
thsowers wrote: post #3

Why so many problems at Google lately? Calendar down two weeks ago[0], and Google Cloud had a larger outage a month ago[1] [0]: https://news.ycombinator.com/item?id=20213092 [1]: https://news.ycombinator.com/item?id=20077421

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside…

[deleted]

Re: Google Cloud networking issues in us-east1

#255

Post by jcims »

ajhurliman wrote: post #252
mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

Seems like if it continues to be a problem that more multi-cloud solutions will present themselves (Terraform does that sort of thing, right?).

Terraform gives you a single management stack to a number of services and endpoints, but it doesn’t magically make your solution multi-cloud...you still need to understand the architecture you are deploying and the idiosyncrasies of each provider and the services used (not a bad thing imo).

Re: Google Cloud networking issues in us-east1

#256

Post by jsjohnst »

dodobirdlord wrote: post #241
jsjohnst wrote: post #229

Earlier quoted context omitted.

In the past ten years: It’s happened more than once with Azure and GCP. I think it happened once with AWS, but not positive there.

AWS had a multi-hour total S3 outage in us-east-1 in February 2017 that knocked out a huge number of things mostly because it turns out that a huge share of their customers run in only 1 region and it's us-east-1. Things mostly continued to work in other regions. I recall Azure had some sort of multi-region database failover disaster that took several regions offline, and GCP has had several global elevated latency/e…

GCP (and all of Google) was down worldwide in 2013 as one example:

https://www.theregister.co.uk/2013/08/17/google_outage/

Here’s one that’s on Azure. Not a 100% total outage like above, but bad enough most I know in the industry would call it being down:

https://www.zdnet.com/article/windows-azure-suffers-worldwid...

If I get a free moment, I’ll dig up other examples, but those were ones that were easy to find.

Re: Google Cloud networking issues in us-east1

#257

Post by laluser »

akhilcacharya wrote: post #251
user5994461 wrote: post #219

Earlier quoted context omitted.

Google employees are commenting publicly and on Hacker News all the time. If there is a policy of not speaking publicly about the company, this has been the most blatantly ignored policy ever.

I’m 90% sure it’s just to flex, honestly. I work at another FANG with a roughly equal engineering community and I don’t see my kind commenting as much at all!

Another FANG = Amazon? If so, Amazon is pretty restrictive in how it wants employees to communicate about internal activities. Most people err on the side of caution and don't comment publicly.

Re: Google Cloud networking issues in us-east1

#258

Post by dgacmu »

m0zg wrote: post #221
Operyl wrote: post #218

Earlier quoted context omitted.

My horribly out of date system works, therefore I should never strive to improve it or god forbid update it (since that involves “fucking with it” in ways that can break it from version to version)? That gets you technical debt and that’s not fun.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…

That's not entirely true. you don't have to try to add features in order for the operating environment of a legacy system to change. More users, transaction count fields overflowing, timestamp fuels hard coated without the century or with 32 bit time_t values...

Or it may simply not meet the needs of users anymore.

I would hardly hold the air traffic control system up as a model to aspire to, for example. The only reason we run the old one is that the upgrade attempts all failed.

Re: Google Cloud networking issues in us-east1

#259

Post by swozey »

ajhurliman wrote: post #252
mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

Seems like if it continues to be a problem that more multi-cloud solutions will present themselves (Terraform does that sort of thing, right?).

microclouds!

Re: Google Cloud networking issues in us-east1

#260

Post by m0zg »

dgacmu wrote: post #258
m0zg wrote: post #221

Earlier quoted context omitted.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…

That's not entirely true. you don't have to try to add features in order for the operating environment of a legacy system to change. More users, transaction count fields overflowing, timestamp fuels hard coated without the century or with 32 bit time_t values... Or it may simply not meet the needs of users anymore. I would hardly hold the air traffic control system up as a model to aspire to, for example. The only re…

Nothing ever is "entirely" true.
Post reply on HN