In the cloud space it seems like AWS does nothing and wins.
Incident Report: Railway Blocked by Google Cloud [resolved]
101–110 of 381 posts
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#102seriously, is it possible to trust GCP with critical data/services at this point if you're not a billion dollar company? I'm exaggerating but someone said they got "auto banned" what if that happens to a small account which hosts some really important data/services there?
Agreed. Railway are probably not far off a billion dollar company though!
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#103Earlier quoted context omitted.
I’m very sympathetic and understand that decisions are easy to criticize in hindsight but leaving your database in GCP while moving everything else to your own data centres seems so backwards I can’t even begin to imagine how that could happen. Was this really an intentional design decision?
> decisions are easy to criticize in hindsight I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project ge…
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#104Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#105Earlier quoted context omitted.
Yea, I mean, that's the whole MO of our platform and we failed at that. So yea, that's disappointing and more so for our customers. I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto…
I’m very sympathetic and understand that decisions are easy to criticize in hindsight but leaving your database in GCP while moving everything else to your own data centres seems so backwards I can’t even begin to imagine how that could happen. Was this really an intentional design decision?
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#106How the heck do these things happen, especially with companies with huge monthly spend? At my last job we had some suspicious workloads running on AWS and our TAM reached out to us before taking any action. Who wants to bet this was some AI automation gone wrong and because GCP seems to be allergic to actually contacting a human to get a response, this just sits in some support queue that outsourced workers look at a…
They all introduce themselves, beg me to setup a meeting w/them and some sort of engineering resource(s), and they come to a meeting with a canned slide deck that is so absurdly unrelated to us that I just laugh, and then the next time I hear from them it's because we have a new AE.
This is my most recent reply (right after Next '26):
> I really appreciate you reaching out; however, we have met with, I dunno at this point, more than a dozen GCP Account reps, execs, technical teams, etc over the years and there's little to no value for us or you, now or in the future. Please do feel free to invest your time on your other clients. We're good; truly.
I love GCP and its services; we have been very pleased with it over the years, but the human side of it? Fucking sucks and I just don't see why they even bother.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#107Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#108Does anyone know how this even happens inside the walls of google? Is it an automated process? How is such a (presumably) high revenue account just magically blocked without human intervention? I'm quite perplexed.
Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation.
Sign up for terms. It will get you payment terms!
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#109How the heck do these things happen, especially with companies with huge monthly spend? At my last job we had some suspicious workloads running on AWS and our TAM reached out to us before taking any action. Who wants to bet this was some AI automation gone wrong and because GCP seems to be allergic to actually contacting a human to get a response, this just sits in some support queue that outsourced workers look at a…
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#110Sorry, I have a hard time blaming Google for this, when Railway seems to be having increasing trouble keeping the platform stable. Something like this should NOT take down an ENTIRE service. There should be a backup when literally your business is about being the reliable backend. This just seems like poor planning to me.
I don't quite know what you mean. Do you really expect Railway to use a multi-cloud architecture to host all of their client's projects? I suspect that would lead to a lower availability, all things considered.