Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

331–340 of 341 posts

Re: Google Cloud networking issues in us-east1

#331

Post by dragonwriter »

archy_ wrote: post #326
m0zg wrote: post #215

Earlier quoted context omitted.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

Cloud should be a backup, a failover, but people build their entire business on other people's hardware because they can sell the cost per hour easier than the price of a new server which is cheaper in the long run. At this point, with so many outages showing the need for self-hosting, not allowing customers to do so shows how little you care about them.

> At this point, with so many outages showing the need for self-hosting,

Are they really showing that? None of the major cloud providers, even constrained to a single region (or even AZ) seems on average less reliable than the on prem datacenters I've seen, and there's

> not allowing customers to do so shows how little you care about them.

While the solutions may not be as complete for all use cases as public-cloud-only ones, are any of the major cloud providers not working to enable and selling their capacity to support hybrid-cloud deployments?

Re: Google Cloud networking issues in us-east1

#332

Post by Doubleslash »

hn_throwaway_99 wrote: post #158
mathattack wrote: post #91

Earlier quoted context omitted.

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

To somewhat echo BurritoElPastor's comment, running a system/app that can be run in multiple clouds is orders of magnitude more difficult than just running a system/app that can be run in multiple regions. And, not to be snarky, but many of the other responses that are along the lines of "It's not really that difficult to run in multiple clouds" - let's just say I have trouble believing these commenters have real wor…

It's not trivial but it's also not an order of magnitude more difficult anymore, as you describe it. There is a reason why Kubernetes gets a lot of backing from corporate customers - precisely because it hides and abstracts most of the underlying infrastructure and provides platform-agnostic primitives that make sense at the application level.

Once you have deployed your stack on Kubernetes, you can pretty much run it on any cloud or infrastructure with minor tweaks at most.

Re: Google Cloud networking issues in us-east1

#333

Post by jodrellblank »

mrweasel wrote: post #316
jodrellblank wrote: post #308

Earlier quoted context omitted.

Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.

I don’t think he’s the one nitpicking. From a business perspective the site was down. Nitpicking is telling him: No it is in fact up, the customer just can’t use it.

"Customers are complaining they can't access a thing"

"OK, we have databases up, load balancers responding, DNS records check out, last change/deployment was at this time, all these services are up, and the latest test suite is running all green, this narrows down the places where a failure might be with some useful differential diagnosis, now we can move attention to.."

"I DON'T CARE THAT YOU THINK THINGS ARE WORKING, IF THE CUSTOMER CANNOT GET TO IT, IT'S DOWN"

"Thanks for that helpful input, let's divert troubleshooting attention from this P1 incident, and have a discussion about what "DOWN" means. You want me to treat the working databases as down because the customer can't get to them? Even though they're working?

It's like the hatred for "works on my machine". "WELL I'M NOT RUNNING ON YOUR MACHINE". No you aren't, but this demonstrates the current build works, the commands you're using are coherent and sensible, excludes many possible causes of failure, and adds useful information to the situation.

Re: Google Cloud networking issues in us-east1

#334

Post by jodrellblank »

shaftoe wrote: post #320
jodrellblank wrote: post #308

Earlier quoted context omitted.

Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.

As someone who lost critical business functionality yesterday when my appengine instances returned only 502s for 5 hours, I find the idea it was "a ridiculous time to nitpick" hilarious. My customers don't care that the network is down, the servers are down, or aliens have landed. The severity is the same and our infrastructure, regardless of the cause, was down. During the impacted time period, we did a full DR fail…

My customers don't care that the network is down, the servers are down, or aliens have landed. The severity is the same and our infrastructure, regardless of the cause, was down.

But the people who have to fix it, desperately care about which specific part is down. That's just about the highest priority information they need. Honing in on where the problem is, is one of the few ways to get to fixing the problem. Having a boss shout that "everything is down, it's all broken" is the opposite of identifying the problem.

find the idea it was "a ridiculous time to nitpick" hilarious.

What? You lost critical business functionality for 5 hours, and you'd rather the boss was shouting at the workers because the wording used doesn't accurately reflect the boss's understanding, instead of the workers working on solving the problem?

Re: Google Cloud networking issues in us-east1

#335

Post by haecceity »

robbiep wrote: post #306
haecceity wrote: post #304

Earlier quoted context omitted.

They only found you because your HN username is same as your Google alias?

woah that's weird. This (Hacceity) is a social media alias of mine. For a moment I thought I wrote this. Did you come across the word in the Mars trilogy too?

Nope. I'm a big fan of scifi. How does haecceity come into Mars?

Re: Google Cloud networking issues in us-east1

#336

Post by hnick »

m0zg wrote: post #215
mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

Have you seen Jonathan's Blow talk that touched on this? I enjoyed it. I think his fundamental point is that as we build on complexity, future generations lose track of the underpinnings and things start failing for unexpected reasons and we may eventually lose our capability entirely. But he does meander a lot.

I've definitely seen this where I work - the "old guard" setup the system that put the company in a prime market position, the newer people are just doing API calls and scratching their heads if it doesn't work.

Here's a reddit link because YouTube is blocked here.

https://www.reddit.com/r/programming/comments/bq1dt6/jonatha...

Re: Google Cloud networking issues in us-east1

#337

Post by m0zg »

bamboozled wrote: post #317
m0zg wrote: post #215

Earlier quoted context omitted.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

So a vulnerability is identified in a version of software you're running within your stack and doing nothing means you will most likely lose important and sensitive customer information if you do nothing about it. Do you: 1) Don't fuck with it? 2) Make a mitigating code change. Patch / fix it (fuck with it)?

Vulnerabilities don't always matter. If it's some godforsaken internal-only backend that never sees external traffic, study whether there's risk, and if there is none, let it be.

If you must fix it, the correct solution is to replace the affected software with the same (or almost the same) version of the software with the fix. No API changes, no other fixes.

Re: Google Cloud networking issues in us-east1

#338

Post by lhoff »

m0zg wrote: post #337
bamboozled wrote: post #317

Earlier quoted context omitted.

So a vulnerability is identified in a version of software you're running within your stack and doing nothing means you will most likely lose important and sensitive customer information if you do nothing about it. Do you: 1) Don't fuck with it? 2) Make a mitigating code change. Patch / fix it (fuck with it)?

Vulnerabilities don't always matter. If it's some godforsaken internal-only backend that never sees external traffic, study whether there's risk, and if there is none, let it be. If you must fix it, the correct solution is to replace the affected software with the same (or almost the same) version of the software with the fix. No API changes, no other fixes.

Sorry, but that's bullshit.

Once an attacker is in your organization he will look for exactly that kind of internal-only backend were exploits are already available and the attack vector is known.

There is no such thing as a internal-only backend regarding security.

Let's assume the attacker used social engineering to get credentials from an unprivileged user and uses these to log in to a remote desktop. (I know there are ways to prevent that but I think there are many examples shown that public facing remote desktop is not two unrealistic) Once he is inside your company he can reach the "internal-only" backend and uses the privilege escalation bug you thought is not worth fixing to get root.

Re: Google Cloud networking issues in us-east1

#339

Post by munificent »

cameronbrown wrote: post #231
munificent wrote: post #227

Earlier quoted context omitted.

I work at Google on an open source project and comment on it frequently. One of the things I really like about working at Google is that they place a lot of trust in the judgement of the individual employees. I generally make it clear when I'm stating my personal opinion versus the "official" (for whatever that means given how informal the project is) one, but I don't have to carefully go through an approved list of…

Off-topic, but I noticed in your bio you wrote Game Programming Patterns. Was a great read!

Thank you!

Re: Google Cloud networking issues in us-east1

#340

Post by robbiep »

haecceity wrote: post #335
robbiep wrote: post #306

Earlier quoted context omitted.

woah that's weird. This (Hacceity) is a social media alias of mine. For a moment I thought I wrote this. Did you come across the word in the Mars trilogy too?

Nope. I'm a big fan of scifi. How does haecceity come into Mars?

The character Sax is asked to describe his belief system, and he says, essentially, that it is haecceity, the this-ness of things, that is his belief system, and I thought that was awesome.

If you haven't read them, you have to!

Post reply on HN