Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

41–50 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#41

Post by fontain »

ndneighbor wrote: post #37
eoswald wrote: post #30

Earlier quoted context omitted.

Yep, and this is why I'm pissed. They lied. They're completely dependent on GCP. So, I gotta do some research, i need something a little more stable (and less dependent on one company's whims) than this. This is bad for them, because it really strikes at the heart of their 'big claim,' peacefull software deployments. This is chaos.

Yea, I mean, that's the whole MO of our platform and we failed at that. So yea, that's disappointing and more so for our customers. I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto…

I’m very sympathetic and understand that decisions are easy to criticize in hindsight but leaving your database in GCP while moving everything else to your own data centres seems so backwards I can’t even begin to imagine how that could happen. Was this really an intentional design decision?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#43

Post by brookst »

impulser_ wrote: post #23
ryanisnan wrote: post #15

Earlier quoted context omitted.

I don't quite know what you mean. Do you really expect Railway to use a multi-cloud architecture to host all of their client's projects? I suspect that would lead to a lower availability, all things considered.

They literally own their own data centers. That's whats surprising about this. They are lying to their customers when they say they operate their own data center because obviously they don't if everyone's apps are down with GCP blocking their account.

Is it not possible that they own their own data center and have an unfortunate Google dependency?

Obviously a fiasco but I’m not prepared to call them liars when it could be an honest mistake.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#44

Post by canpan »

rekabis wrote: post #17

TL;DR: putting all your eggs into one basket is bad, man.

How to handle domains? The rest is easy, but your domain registrar blocking you sounds like a pain. My current solution is to use a local small provider, just for the domain. Then if there is a problem with your play account it is out of any blast radius.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#45

Post by irjustin »

eoswald wrote: post #27
ryanisnan wrote: post #15

Earlier quoted context omitted.

I don't quite know what you mean. Do you really expect Railway to use a multi-cloud architecture to host all of their client's projects? I suspect that would lead to a lower availability, all things considered.

Well, in the same token, is it smart to base your ENTIRE architecture on a single cloud architecture? Isn't that why some of us build in fallbacks for AWS-hosted services? I mean, their enitre platform, both public and private facing, is running on the same thing. One error, one problem, takes out the entire service.

Taking this at face value, this doesn't happen to AWS clients - at least I don't read about it here.

AWS may have data centers[0] go[1] down[2], but that's within expected bounds of standard ops.

[0] https://hooks.slack.com/services/TJ7HQS7FC/B0B5S7UTBJ4/PUHIC...

[1] https://www.aljazeera.com/news/2025/10/21/what-caused-amazon...

[2] https://netflixtechblog.com/lessons-netflix-learned-from-the...

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#46

Post by add-sub-mul-div »

faangguyindia wrote: post #4

Google cloud also locked out a Korean Goverment Organization recently. The guy posted on GCP subreddit. Google really need to improve their support team. It's strange such a big corp can't even afford to have proper support team.

Automating support, automating everything is the key to their whole deal. Tech giants leapfrogged the rest of the economy by innovating a company that can scale its customers without having to scale itself proportionally.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#47

Post by TheTaytay »

I’ve seen a few smug “all your eggs in one basket” comments here.

I’m aware of some companies hosting their own metal and infra, but I’m not aware of large companies mitigating risk by hosting on separate cloud providers as a fallback mechanism. We might disagree with cloud provider choice, or think they should have been hosting their own metal, but that’s still an “all your eggs in one basket” choice, right?

Heck, they might even have multi-region fallback with GCP, but if GCP bans your account, that doesn’t matter.

Are there good examples of running a company of railway’s size so redundantly that their host could nuke one of their accounts and they’d just keep on trucking?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#49

Post by fontain »

TheTaytay wrote: post #47

I’ve seen a few smug “all your eggs in one basket” comments here. I’m aware of some companies hosting their own metal and infra, but I’m not aware of large companies mitigating risk by hosting on separate cloud providers as a fallback mechanism. We might disagree with cloud provider choice, or think they should have been hosting their own metal, but that’s still an “all your eggs in one basket” choice, right? Heck, t…

They do run their own metal. That’s their entire ethos. Railway is their own cloud.
Post reply on HN