This could kill a startup. I really don't like Google's automated and silent account murder functionality.
Incident Report: Railway Blocked by Google Cloud [resolved]
51–60 of 381 posts
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#52This is bad. Even their own website is down at railway.com. Looks like total dependency on google cloud. Surprising for a company of their scale with all this VC money.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#53Earlier quoted context omitted.
Yea, I mean, that's the whole MO of our platform and we failed at that. So yea, that's disappointing and more so for our customers. I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto…
I’m very sympathetic and understand that decisions are easy to criticize in hindsight but leaving your database in GCP while moving everything else to your own data centres seems so backwards I can’t even begin to imagine how that could happen. Was this really an intentional design decision?
> decisions are easy to criticize in hindsight
I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project get deleted just like that.
(Even if we did get assurances from them in 2024, that it wouldn't happen again, although we just got auto-rate limited the last time.)
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#54TL;DR: putting all your eggs into one basket is bad, man.
How to handle domains? The rest is easy, but your domain registrar blocking you sounds like a pain. My current solution is to use a local small provider, just for the domain. Then if there is a problem with your play account it is out of any blast radius.
What do you mean by local small provider? A registrar on main street?
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#55[deleted]
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#56Earlier quoted context omitted.
They literally own their own data centers. That's whats surprising about this. They are lying to their customers when they say they operate their own data center because obviously they don't if everyone's apps are down with GCP blocking their account.
Is it not possible that they own their own data center and have an unfortunate Google dependency? Obviously a fiasco but I’m not prepared to call them liars when it could be an honest mistake.
I imagine there's also an important difference between:
1. We depend on X but could gracefully migrate to an alternate in a week if we really needed to.
2. All data is mirrored instantly so that we can do seamless fail-over in case X has its own outage.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#57Google cloud also locked out a Korean Goverment Organization recently. The guy posted on GCP subreddit. Google really need to improve their support team. It's strange such a big corp can't even afford to have proper support team.
> It's strange such a big corp can't even afford to have proper support team Railway say they are in touch with that support team.
god help them
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#58It is killing me...
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#59It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.
AWS normally contacts you first.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#60github got way more noise for less