Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

311–320 of 341 posts

Re: Google Cloud networking issues in us-east1

#311

Post by why_only_15 »

m0zg wrote: post #260
dgacmu wrote: post #258

Earlier quoted context omitted.

That's not entirely true. you don't have to try to add features in order for the operating environment of a legacy system to change. More users, transaction count fields overflowing, timestamp fuels hard coated without the century or with 32 bit time_t values... Or it may simply not meet the needs of users anymore. I would hardly hold the air traffic control system up as a model to aspire to, for example. The only re…

Nothing ever is "entirely" true.

Of course - the point this person was making is that this is substantially false in the way they described.

Re: Google Cloud networking issues in us-east1

#312

Post by Voxoff »

jodrellblank wrote: post #308
mrweasel wrote: post #303

Earlier quoted context omitted.

>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.

Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.

"to shout and add stress to an already difficult situation" now that's accuracy

Re: Google Cloud networking issues in us-east1

#313

Post by mitchs »

zamadatix wrote: post #205
fredthomsen wrote: post #121

Earlier quoted context omitted.

my brother-in-law's construction company actually did just that. ground wasn't properly marked and the fiber got cut, multiple links

It's not uncommon to see 500 strand in one tube get cut by a backhoe. So much so it's even jargon at this point http://www.catb.org/jargon/html/F/fiber-seeking-backhoe.html

500? Those are rookie numbers.

Re: Google Cloud networking issues in us-east1

#314

Post by titzer »

owenmarshall wrote: post #235
m0zg wrote: post #221

Earlier quoted context omitted.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…

I got an email yesterday that told me the boring old HPUX server (which was racked before my intern was even born) barfed all over its boring old 50-pin SCSI drive and ops went scrambling to find one in storage so the boring old Oracle DB that was responsible for production lines running could be recovered. Took us around an hour, cost us a boring 5 figures. Luckily our sysadmin knows how to hide “unused parts” for d…

> you have to do something to fight the entropy.

Stuff breaks. So you fix it. Boring old stuff needs fixing too sometimes. Problem is, old stuff gets obsolete, can't get replacement parts, because of progress. (or something). It's the same story since the first looms were made centuries ago.

What you can't fix you can't really depend on. Our time scales are just compressed to ridiculousness because the pace of change is off the charts these days. So basically, you can't really depend on anything working more than a few months before falling over. Sucks.

Re: Google Cloud networking issues in us-east1

#315

Post by issati »

hajhatten wrote: post #307
harshreality wrote: post #143

Hacker News: The real status page and help desk for the internet. Do companies realize how absurd this is? ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.

This reminds me of an incident in Sweden a couple of years ago. We test our disaster alarms on a known schedule. And just a couple of years ago, during the peak vacation time in the summer, the alarm went off, off schedule. This made the entire country panic. Were we being attacked? The agency that is supposed to let people know through the public channels like tv, radio etc were silent. They were themselves on vacat…

It goes to show how badly it is set up for a false alarm. In a real emergency all the primary functions would go up (taking over radio broadcasts for example) so there wouldn't be the same problem. It is still bad of course because of the "cry wolf" factor.

Re: Google Cloud networking issues in us-east1

#316

Post by mrweasel »

jodrellblank wrote: post #308
mrweasel wrote: post #303

Earlier quoted context omitted.

>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.

Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.

I don’t think he’s the one nitpicking. From a business perspective the site was down. Nitpicking is telling him: No it is in fact up, the customer just can’t use it.

Re: Google Cloud networking issues in us-east1

#317

Post by bamboozled »

m0zg wrote: post #215
mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

So a vulnerability is identified in a version of software you're running within your stack and doing nothing means you will most likely lose important and sensitive customer information if you do nothing about it.

Do you:

1) Don't fuck with it?

2) Make a mitigating code change. Patch / fix it (fuck with it)?

Re: Google Cloud networking issues in us-east1

#318

Post by bardworx »

Thorrez wrote: post #275
bardworx wrote: post #213

Earlier quoted context omitted.

That’s...that’s some petty fucking shit. I didn’t go through your comments but considering your email is in your profile, someone really had to have a hard-on to report you for leaks. I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN (when applied in a normal circumstance of adding additional information or correcting a…

You're assuming that the person making the report said "boulos needs to be fired!". Maybe the person making the report said "Hey, I found some internal details on this external site. I'm not sure if this is allowed. Maybe someone who knows more should take a look at it, here's the link to the page."

Their email is in their profile. I would think it is sensible to reach out to them directly or speak with your manager to get a second opinion.

Submitting a complaint to an internal review because “you’re not sure it’s allowed” is really petty.

In my opinion, and experience, folks who have good intentions usually pull you to the side to get a feel for a situation before filing a formal complaint.

Re: Google Cloud networking issues in us-east1

#319

Post by bardworx »

joshuamorton wrote: post #281
praneshp wrote: post #268

Earlier quoted context omitted.

> I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN This is not so difficult though. You just need to adjust your starting point to someone who doesn't like boulos' first. That's not so difficult IMO, it's a large org and boulos' seems to be a fairly prolific commenter here.

It also could be someone will intentioned who believes boulos is sharing things he shouldn't be. He certainly shares stuff I wouldn't be comfortable sharing, but then again he's a lot better connected and in the know than I am.

If you are their co-worker and believe he shared some info that shouldn’t be public, wouldn’t it be a simple curtesy to email them and get some clarity? That seems like a reasonable thing to do.

On the other hand, to anonymously submit a complaint feels, to me, like a personal attack. Someone who simply doesn’t like them in for whatever reason. To me, that action seem petty.

Re: Google Cloud networking issues in us-east1

#320

Post by shaftoe »

jodrellblank wrote: post #308
mrweasel wrote: post #303

Earlier quoted context omitted.

>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.

Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.

As someone who lost critical business functionality yesterday when my appengine instances returned only 502s for 5 hours, I find the idea it was "a ridiculous time to nitpick" hilarious.

My customers don't care that the network is down, the servers are down, or aliens have landed. The severity is the same and our infrastructure, regardless of the cause, was down.

During the impacted time period, we did a full DR failover to appengine instances we spun up in west2. This was not a minor hiccup.

Post reply on HN