Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

61–70 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#61
post #44
post #17

TL;DR: putting all your eggs into one basket is bad, man.

How to handle domains? The rest is easy, but your domain registrar blocking you sounds like a pain. My current solution is to use a local small provider, just for the domain. Then if there is a problem with your play account it is out of any blast radius.

MarkMonitor

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#62

I’ve seen a few smug “all your eggs in one basket” comments here. I’m aware of some companies hosting their own metal and infra, but I’m not aware of large companies mitigating risk by hosting on separate cloud providers as a fallback mechanism. We might disagree with cloud provider choice, or think they should have been hosting their own metal, but that’s still an “all your eggs in one basket” choice, right? Heck, t…

Just google multi-cloud. Yes. It's a thing.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#63

This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their scale with all this VC money.

They run a decent amount of their own compute/bare metal server for customer workloads. But likely still had some critical dependencies on GCP.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#64
post #41

Earlier quoted context omitted.

Yea, I mean, that's the whole MO of our platform and we failed at that. So yea, that's disappointing and more so for our customers. I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto…

I’m very sympathetic and understand that decisions are easy to criticize in hindsight but leaving your database in GCP while moving everything else to your own data centres seems so backwards I can’t even begin to imagine how that could happen. Was this really an intentional design decision?

I have exactly the same architecture. You can easily administer a postgres/mysql on your own infrastructure, but it's also the one thing where backups and availability are super strict. I can easily support multi-region in Google Cloud or AWS and that's way harder to do on-prem, and it's also hard to handle the replication story as safely as with Google Cloud. The hope is that GCP et al. give you safety and availability for the control plane stuff and you can run your data plane on-prem.

At $2m/mo spend, this kind of thing is insane. GCP has never been the most reliable of clouds but this is pretty awful. I would never have expected this.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#66

Google cloud also locked out a Korean Goverment Organization recently. The guy posted on GCP subreddit. Google really need to improve their support team. It's strange such a big corp can't even afford to have proper support team.

pretty sure their support team is a flaky ML model that is haplessly flagging random accounts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#68
seriously, is it possible to trust GCP with critical data/services at this point if you're not a billion dollar company?

I'm exaggerating but someone said they got "auto banned"

what if that happens to a small account which hosts some really important data/services there?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#69

Google cloud also locked out a Korean Goverment Organization recently. The guy posted on GCP subreddit. Google really need to improve their support team. It's strange such a big corp can't even afford to have proper support team.

Not strange, Google has never had a proper support team unless you are an "Enterprise" level customer.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#70
post #41

Earlier quoted context omitted.

I’m very sympathetic and understand that decisions are easy to criticize in hindsight but leaving your database in GCP while moving everything else to your own data centres seems so backwards I can’t even begin to imagine how that could happen. Was this really an intentional design decision?

> decisions are easy to criticize in hindsight I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project ge…

could you clarify, did an automated process by Google delete a GCP project/account/resource(s)? like, what exactly were you seeing when trying to get access or see what happened?
Post reply on HN