Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

261–270 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#261

Post by rekabis »

canpan wrote: post #44
rekabis wrote: post #17

TL;DR: putting all your eggs into one basket is bad, man.

How to handle domains? The rest is easy, but your domain registrar blocking you sounds like a pain. My current solution is to use a local small provider, just for the domain. Then if there is a problem with your play account it is out of any blast radius.

What the deuce are you blathering on about. An account got blocked, this has nothing to do with a domain.

And I’m talking about having disparate failovers that don’t rely on a single hosting provider. At that point, who cares what Google does to your cloud account… work with the hot failover and spin up another hot failover somewhere else.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#262

Post by karlkloss »

valgaze wrote: post #182

May 2024 UniSuper incident: https://cloud.google.com/blog/products/infrastructure/detail... https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... A joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian 8 May 2024 UniSuper and Google Cloud understand the disruption to services experienced by members has been extremely frustrating and disappointing. We extend our sincere apologies to…

"deletion of the Private Cloud subscription triggered deletion across both geographies"

It's called single point of failure, and it's the nightmare of everyone who was ever in charge of safety.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#263

Post by locknitpicker »

tardwrangler wrote: post #248

Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.

> Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence.

This. It's very odd that in other threads we see a bunch of accounts heavily invested in criticizing a cloud provider, but what's conspicuously absent from this wave of indignation is any curiosity in the root cause, or even any interest in exploring what it might have been. Quite odd.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#264

Post by rekabis »

binarycleric wrote: post #98
rekabis wrote: post #17

TL;DR: putting all your eggs into one basket is bad, man.

Same applies to all the companies betting the farm on AWS.

Precisely. If you’re going to have a hot failover, it behoves you to have an entirely separate entity billing you for that hosting.

Honestly, I don’t know where the downvotes are coming from. Do people have no clue about service resiliency? I can understand if it’s a personal project or you haven’t yet scaled to paying customers, but anything at scale with serious money involved needs to be completely independent of the underlying hosting. It should remain up even if an entire provider goes titsup.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#265

Post by jameson »

__s wrote: post #129
jameson wrote: post #95

Earlier quoted context omitted.

What was the reason GCP took down a startup previously?

hn.algolia.com gcp blocked https://news.ycombinator.com/item?id=46731498 https://news.ycombinator.com/item?id=33360416 Then I recall https://news.ycombinator.com/item?id=45798827 https://news.ycombinator.com/item?id=33737577

Wow... Just wow...

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#266

Post by rmeara »

codegeek wrote: post #52

This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their scale with all this VC money.

Google has a total dependency on it's own infra and does fine. Why do its customers need multicloud? Huge PITA unless you need an absurd number of 9s

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#267

Post by jedberg »

Alive-in-2025 wrote: post #222
shye wrote: post #204

Earlier quoted context omitted.

Apple also uses AWS, and I won't be surprised if they also use Azure. Big companies are multicloud, and not because it's a good idea (it rarely is), but because they inherited multiple environments on different CSPs, and maintaining those where they are is often cheaper than migrating them to a different CSP.

I wonder if big companies can get a special contract with something like you can't delete my service automatically (unless it's an emergency)

If you're big enough you don't need a contract for that, that's just their default method of operation.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#268

Post by buildbot »

VirusNewbie wrote: post #258
adamtaylor_13 wrote: post #174

Earlier quoted context omitted.

Perhaps you don't notice GCP outages because so few companies rely on them?

Spotify, Ebay, Paypal, Apple, Walmart, Uber are huge users. Lots of other big named companies are big users that I don't think are public. Then there's Anthropic... huge user.

Snap, for like 15 years at this point: https://cloud.google.com/blog/products/gcp/snap-maintains-up...

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#269

Post by suttontom »

dantiberian wrote: post #221
valgaze wrote: post #182

May 2024 UniSuper incident: https://cloud.google.com/blog/products/infrastructure/detail... https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... A joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian 8 May 2024 UniSuper and Google Cloud understand the disruption to services experienced by members has been extremely frustrating and disappointing. We extend our sincere apologies to…

I wrote about the UniSuper issue at the time: https://danielcompton.net/google-cloud-unisuper . It was a pretty nasty bug where their VMWare environment was created with a one-year expiry date, but was one "resource" from the perspective of Google Cloud.

"UniSuper’s production Google Cloud VMware Engine (GCVE) private cloud was automatically deleted one year after it’s creation due to a misconfiguration in how it was created. When it was created, there was a bug in the creation script which passed a null value."

That's pretty amazing. Not due to a cascading failure from someone changing a config deep inside of a system that caused a bunch of unintended effects, just someone who messed up writing a shell script?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#270

Post by raverbashing »

dantiberian wrote: post #221
valgaze wrote: post #182

May 2024 UniSuper incident: https://cloud.google.com/blog/products/infrastructure/detail... https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... A joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian 8 May 2024 UniSuper and Google Cloud understand the disruption to services experienced by members has been extremely frustrating and disappointing. We extend our sincere apologies to…

I wrote about the UniSuper issue at the time: https://danielcompton.net/google-cloud-unisuper . It was a pretty nasty bug where their VMWare environment was created with a one-year expiry date, but was one "resource" from the perspective of Google Cloud.

Creating stuff with 1yr (implicit) expiry by default is just a delayed footgun tbh
Post reply on HN