Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

321–330 of 341 posts

Re: Google Cloud networking issues in us-east1

#321

Post by owenmarshall »

kalleboo wrote: post #285
owenmarshall wrote: post #235

Earlier quoted context omitted.

I got an email yesterday that told me the boring old HPUX server (which was racked before my intern was even born) barfed all over its boring old 50-pin SCSI drive and ops went scrambling to find one in storage so the boring old Oracle DB that was responsible for production lines running could be recovered. Took us around an hour, cost us a boring 5 figures. Luckily our sysadmin knows how to hide “unused parts” for d…

There's boring and there's legacy. Legacy is when the hardware is unsupported and doesn't get software/security updates. You don't want to let it become legacy. The counter-story to yours is running that database on MongoDB in the cloud on a cluster. Instead you'd be having crazy MongoDB issues, data inconsistencies, connectivity issues when the cloud is down, etc etc. The solution is somewhere in the middle. You can…

I think you are right. Where I’ve seen success in “the boring middle” is when an appropriate amount of tension exists in the engineering organization: you want some teams and groups pushing to try new things, but they need to push against a boundary - preferably something, not someone - and the boundary should define your organization’s best practices and standards. This way a team doesn’t get to sneak clustered Mongo into the cloud and make your ordering systems talk to it.

But over time boring IT turns into legacy, and without some tension to the system pushing it forward your standards end up locking you into legacy forever.

Re: Google Cloud networking issues in us-east1

#322

Post by owenmarshall »

m0zg wrote: post #262
owenmarshall wrote: post #235

Earlier quoted context omitted.

I got an email yesterday that told me the boring old HPUX server (which was racked before my intern was even born) barfed all over its boring old 50-pin SCSI drive and ops went scrambling to find one in storage so the boring old Oracle DB that was responsible for production lines running could be recovered. Took us around an hour, cost us a boring 5 figures. Luckily our sysadmin knows how to hide “unused parts” for d…

The first part of your post sounds like a success story to me. You got many years of use out of that server. 5 figures is cheap comparing to the cost (including the collateral damage cost) of a hotshot SWE or devops guy who insists on using the most resume-worthy, most bleeding-edge technology available.

I wish I could upvote you twice - I just finished a multi-week effort to unwind (defuse?) some of the resume-driven architectures that were left behind when resume-driven development was successful.

Re: Google Cloud networking issues in us-east1

#323

Post by tyingq »

ti_ranger wrote: post #299
jsjohnst wrote: post #230

Earlier quoted context omitted.

All of S3 has, but that’s because S3 had a single choke point in a single region for a long time.

> All of S3 has, but that’s because S3 had a single choke point in a single region for a long time. The only S3 event here was limited to us-east-1: https://aws.amazon.com/premiumsupport/technology/pes/ Some APIs were impacted, because they are global by nature (e.g create-bucket). But S3 was working fine in all other regions, for existing buckets. However, many websites were affected, because they didn't use any of…

All of S3 create-bucket is "all of S3" for a lot of use cases and customers.

Re: Google Cloud networking issues in us-east1

#324

Post by owenmarshall »

hdfbdtbcdg wrote: post #297
rrdharan wrote: post #272

Earlier quoted context omitted.

Curious how you would imagine handling something like GDPR or SOX compliance in this alternative world you’re proposing. You can’t magically foresee new requirements and new implied complexity for all future time.

To many European companies GDPR did not really change the operational requirements - only the penalties for not meeting them.

That’s too clever by half. Avoiding substantial financial penalties for not meeting an operational requirement is an operational requirement.

Re: Google Cloud networking issues in us-east1

#325

Post by archy_ »

issati wrote: post #315
hajhatten wrote: post #307

Earlier quoted context omitted.

This reminds me of an incident in Sweden a couple of years ago. We test our disaster alarms on a known schedule. And just a couple of years ago, during the peak vacation time in the summer, the alarm went off, off schedule. This made the entire country panic. Were we being attacked? The agency that is supposed to let people know through the public channels like tv, radio etc were silent. They were themselves on vacat…

It goes to show how badly it is set up for a false alarm. In a real emergency all the primary functions would go up (taking over radio broadcasts for example) so there wouldn't be the same problem. It is still bad of course because of the "cry wolf" factor.

I think a similar situation happened in Hawaii last year and it took awhile to send a false alert message.

Re: Google Cloud networking issues in us-east1

#326

Post by archy_ »

m0zg wrote: post #215
mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

Cloud should be a backup, a failover, but people build their entire business on other people's hardware because they can sell the cost per hour easier than the price of a new server which is cheaper in the long run. At this point, with so many outages showing the need for self-hosting, not allowing customers to do so shows how little you care about them.

Re: Google Cloud networking issues in us-east1

#327

Post by [deleted] »

akhilcacharya wrote: post #251
user5994461 wrote: post #219

Earlier quoted context omitted.

Google employees are commenting publicly and on Hacker News all the time. If there is a policy of not speaking publicly about the company, this has been the most blatantly ignored policy ever.

I’m 90% sure it’s just to flex, honestly. I work at another FANG with a roughly equal engineering community and I don’t see my kind commenting as much at all!

[deleted]

Re: Google Cloud networking issues in us-east1

#328

Post by [deleted] »

jodrellblank wrote: post #308
mrweasel wrote: post #303

Earlier quoted context omitted.

>The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.

Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.

[deleted]

Re: Google Cloud networking issues in us-east1

#329

Post by KirinDave »

sudosteph wrote: post #253
david-cako wrote: post #164

Earlier quoted context omitted.

I work for AWS. There is typically a balance that has to be struck when sharing information with customers. I would imagine this goes for most companies, which is why it isn't until a post-mortem that the messaging is fully refined.

True, but I'd argue that the "Customer Obsession" priciple would drive you to attempt some sort of good-faith effort towards real-time communication. Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why…

I'm late to this party but I just wanna add, boulos could be wrong or inaccurate and it wouldn't be a big deal. Those status updates are communications to customers, and thus tend to be more conservative. Inaccuracy is a much bigger deal there than the informal status here.

Sadly, the closer you are to the action of a thing like this (for example, I'm on NetInfra SRE and we were part of the group that put in place the current mitigations you're seeing work now), the less you can say without fear of subtle inaccuracy or releasing non-public information.

Re: Google Cloud networking issues in us-east1

#330

Post by KirinDave »

archy_ wrote: post #326
m0zg wrote: post #215

Earlier quoted context omitted.

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding. I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are…

Cloud should be a backup, a failover, but people build their entire business on other people's hardware because they can sell the cost per hour easier than the price of a new server which is cheaper in the long run. At this point, with so many outages showing the need for self-hosting, not allowing customers to do so shows how little you care about them.

The challenge of self hosting is then you don't get the sophisticated load balancing the CSPs like my employer, Microsoft and Amazon offer. You also don't get the dedicated networks.

But it's true, it's much cheaper if you can find a way to replicate those or do without.

Post reply on HN