Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

201–210 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#201

Post by modernpacifist »

kvakvs wrote: post #197
valgaze wrote: post #182

May 2024 UniSuper incident: https://cloud.google.com/blog/products/infrastructure/detail... https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... A joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian 8 May 2024 UniSuper and Google Cloud understand the disruption to services experienced by members has been extremely frustrating and disappointing. We extend our sincere apologies to…

The instant cascading worldwide deletion upon closing or deleting a subscription sounds like a recipe for disaster. Why not mark it for deletion and delete say... a day or a week later?

Either mark-for-delete has the same impact as deleting in terms of shooting all the Cloud resources associated with the subscription, at which point the outage still happens but maybe the recovery is smoother or you've just delayed the inevitable by a week because no one will look at it unless there is actual impact.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#203

Post by shye »

kvakvs wrote: post #197
valgaze wrote: post #182

May 2024 UniSuper incident: https://cloud.google.com/blog/products/infrastructure/detail... https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... A joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian 8 May 2024 UniSuper and Google Cloud understand the disruption to services experienced by members has been extremely frustrating and disappointing. We extend our sincere apologies to…

The instant cascading worldwide deletion upon closing or deleting a subscription sounds like a recipe for disaster. Why not mark it for deletion and delete say... a day or a week later?

From personal experience, as a customer who once did something stupid: Google Cloud does soft deletes. But you need to reach out to support fast enough. And really, if you deleted something important and discovered it only the next day, and not within minutes, you're having a bigger issue that a soft delete won't solve.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#204

Post by shye »

fragmede wrote: post #189
adamtaylor_13 wrote: post #174

Earlier quoted context omitted.

Perhaps you don't notice GCP outages because so few companies rely on them?

GCP has a lot of customers. But you wouldn't know the companies that do, unless you worked there and wanted to leak it, or it publicly comes out. Eg it's been publicly acknowledged that Apple uses GCP for iCloud, https://www.cnbc.com/amp/2018/02/26/apple-confirms-it-uses-g... , and Home Depot is another that's used as a case study, https://cloud.google.com/customers/the-home-depot but most customers don't want to mak…

Apple also uses AWS, and I won't be surprised if they also use Azure. Big companies are multicloud, and not because it's a good idea (it rarely is), but because they inherited multiple environments on different CSPs, and maintaining those where they are is often cheaper than migrating them to a different CSP.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#206

Post by hasyimibhar »

erikerikson wrote: post #192
andreareina wrote: post #184

Earlier quoted context omitted.

Didn't the latest outage affect people not on us-east-1 because internal aws services depend on us-east-1?

The impacts are usually partial. For example, scaling is impacted but everything already deployed contributes to work up to capacity. Or, you can't change configuration but the previous configuration works as configured. Often surprisingly not so impactful even if there can be limited work stoppage.

The problem with the us-east-1 outage is that a lot of big companies are there, so even if you try your best not to depend on us-east-1, your third party providers are most likely there. In my previous company, we were completely down during us-east-1 outage because of other dependencies that are beyond our control.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#209

Post by manapause »

kvakvs wrote: post #197
valgaze wrote: post #182

May 2024 UniSuper incident: https://cloud.google.com/blog/products/infrastructure/detail... https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... A joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian 8 May 2024 UniSuper and Google Cloud understand the disruption to services experienced by members has been extremely frustrating and disappointing. We extend our sincere apologies to…

The instant cascading worldwide deletion upon closing or deleting a subscription sounds like a recipe for disaster. Why not mark it for deletion and delete say... a day or a week later?

It’s a good question. That said unless there are compliance or fallback concerns i would prefer a service that burns my data on departure.
Post reply on HN