Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

231–240 of 341 posts

Re: Google Cloud networking issues in us-east1

#231

Post by cameronbrown »

munificent wrote: post #227
ricardobeat wrote: post #193

Earlier quoted context omitted.

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

I work at Google on an open source project and comment on it frequently. One of the things I really like about working at Google is that they place a lot of trust in the judgement of the individual employees. I generally make it clear when I'm stating my personal opinion versus the "official" (for whatever that means given how informal the project is) one, but I don't have to carefully go through an approved list of…

Off-topic, but I noticed in your bio you wrote Game Programming Patterns. Was a great read!

Re: Google Cloud networking issues in us-east1

#232

Post by hunter2_ »

hinkley wrote: post #170
jsty wrote: post #141

Earlier quoted context omitted.

However much we technical people might salivate at the prospect of designing a multi-cloud solution, for the vast majority of businesses it simply isn't worth the cost / complexity. I'd wager 90-something percent of applications could suffer multi-hour outages without impacting business function to any measurable degree. Plus the fact that without serious investment, you're probably more liable to decrease availabili…

The real trick here, which many people don’t want to look at, is to avoid overly centralizing your workflow. I can get a lot of work done while Outlook is down. Hell, probably more work done. If our build server is down I can work for a couple hours (unless we’ve done something very bad). Same for git or our bug database or wiki or or or. When I get stuck on one thing I can swap to something else every couple of hour…

Probably lost 1 of 3 phases. You're quite right in that the decision of what phase a circuit is on has a lot to do with business, and hopefully no major repurposing of the space without rewiring the space has occurred. For lighting, you'd want 1/3 of fixtures per room to go out, not 1/3 of your rooms in their entirety. For appliances and receptacles, you'd rather lose a whole function (the kitchen) than be able to cook but not do dishes, with every function trying to figure out oddball workarounds.

Re: Google Cloud networking issues in us-east1

#233

Post by walshemj »

boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

Sounds like back hoe fade (from the write up) and it sounds like multiple cables sharing the same physical route got taken out.

Re: Google Cloud networking issues in us-east1

#234

Post by jjeaff »

toomuchtodo wrote: post #187
opportune wrote: post #124

Earlier quoted context omitted.

Proprietary managed services can save a lot of dev/setup/SRE time though. Many businesses have more pressing things to work on than spending dev time to prevent vendor lock-in.

Everyone spends their runway differently. Once you’re off the ground, derisk.

Most companies don't have a "runway", they are just bootstrapped and have to actually justify their expenses and lock-in every day.

Re: Google Cloud networking issues in us-east1

#235

Post by owenmarshall »

m0zg wrote: post #221
Operyl wrote: post #218

Earlier quoted context omitted.

My horribly out of date system works, therefore I should never strive to improve it or god forbid update it (since that involves “fucking with it” in ways that can break it from version to version)? That gets you technical debt and that’s not fun.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…

I got an email yesterday that told me the boring old HPUX server (which was racked before my intern was even born) barfed all over its boring old 50-pin SCSI drive and ops went scrambling to find one in storage so the boring old Oracle DB that was responsible for production lines running could be recovered. Took us around an hour, cost us a boring 5 figures. Luckily our sysadmin knows how to hide “unused parts” for days like that or we’d have been really in trouble.

> You can’t create “technical debt” if you don’t change anything in the first place.

Rubbish. The bits really do rot, and if you don’t do _something_ on occasion you end up with an entire data center no one wants to touch because the dust in the servers might be structural at this point.

I’m not saying go rewrite your apps against the Kafka instance your junior devs are fucking with, but you have to do something to fight the entropy.

Re: Google Cloud networking issues in us-east1

#236

Post by danso »

cameronbrown wrote: post #231
munificent wrote: post #227

Earlier quoted context omitted.

I work at Google on an open source project and comment on it frequently. One of the things I really like about working at Google is that they place a lot of trust in the judgement of the individual employees. I generally make it clear when I'm stating my personal opinion versus the "official" (for whatever that means given how informal the project is) one, but I don't have to carefully go through an approved list of…

Off-topic, but I noticed in your bio you wrote Game Programming Patterns. Was a great read!

Also off-topic: am looking forward to the finishing of craftinginterpreters.com, which has been a fantastic read so far

Re: Google Cloud networking issues in us-east1

#237

Post by dodobirdlord »

m0zg wrote: post #215
mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

> 99.99% of all these outages are due to screwing up something that already works

Fiber optic cables are a great technology, but they don't react well to being cut in half by a backhoe. Is the solution you are recommending that we stop using fiber optic cables, or that we stop using backhoes?

Re: Google Cloud networking issues in us-east1

#238

Post by jjeaff »

stingraycharles wrote: post #112
opportune wrote: post #94

Earlier quoted context omitted.

Not him but oftentimes cloud outages can be due to issues with the network connections to the datacenter, or power outages. Datacenters also sometimes have other single points of failure such as DNS, but those are within the company's control. https://www.networkworld.com/article/3373646/network-problem... https://www.datacenterknowledge.com/uptime/equinix-power-out...

But data centers are typically designed with network and power failures in mind, not? Isn’t this why these kind of ring based network topologies exist, so that whenever a single network connection fails, it can still easily be routed around?

Almost always, yes, but the problem is that everyone has to start routing around the problem and it creates congestion. Those redundant pipes don't sit idle. They are sharing the traffic.

As mentioned in another thread, in this case, Google has rerouted google.com traffic out of the region to try to mitigate the congestion.

Re: Google Cloud networking issues in us-east1

#239

Post by fredthomsen »

niyazpk wrote: post #228
mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

As more businesses move their compute to the cloud, one might predict that more people will be impacted by outages in the large cloud providers. This in turn means that the affected people will start up-voting these threads. Expect these to be more common.

And unfortunately that is making the web more centralized.
Post reply on HN