Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

311–320 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#311

Post by graemep »

JoRyGu wrote: post #134
somewhatgoated wrote: post #115

Earlier quoted context omitted.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

AWS goes down catastrophically but are back up in minutes/hours most of the time (as long as they aren't down because Iran blew up their data center). That's obviously REALLY bad for certain industries, but I suspect for the vast majority of their customers it's not a big deal. We've been able to isolate the damage almost every time just by having AZ failover in place and avoiding us-east-1 where we can.

Failover is supposed to protect you every time, unless something really exceptional happens.

While its possible to to isolate the effects, judging by how many things stop working when there is an AWS failure a lot of people fail to do that. I think the shit of responsibility to AWS removes the incentive to put effort into resilience against AWS failure.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#313

Post by darkwater »

realityking wrote: post #290
garciasn wrote: post #141

Earlier quoted context omitted.

Before I just cut them off entirely, I used to tell them my primary concern was cost savings and that I wanted them to recommend ways I could cut 25% off my bill every month and watch the glorified salespeople fumble over trying to avoid that conversation. It’s ok though, Claude helped us cut >45% of our monthly costs. I’m surprised they haven’t been beating down my door after we made that level-shift. Probably in AE…

My experience with a large-ish ($5m/year) AWS account was quite different. They were happy to support us with cost optimizations, discounts, and one time credits for certain activities (co-innovation and archiving certain milestones in their partner program). Their primary concern seemed to have been to keep as much of our workload inside AWS as possible and to win workload from 3rd party services we used (e.g. CDNs)…

This is my experience as well with AWS accounts in the 2-4M/year range, the biggest upsell they always try to make is Enterprise Support, but for the rest they are usually happy to help you cut cost in the short term - as long as you stay with them in the long term.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#314

Post by maipen »

tardwrangler wrote: post #248

Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.

Two years ago I needed their support and they were so toxic that I just moved to vercel and told them to f off. But I wanted something similar for other services and then I found coolify. There’s absolutely no reason to use railway when you can use coolify.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#315

Post by [deleted] »

tardwrangler wrote: post #248

Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.

[deleted]

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#316

Post by x0x0 »

jeffreyq wrote: post #257
prathamtharwani wrote: post #250

Earlier quoted context omitted.

Could you point us to any specific past instances? I'd be interested to read about them.

https://blog.railway.com/p/incident-report-february-11-2026

"we did not have the monitoring or controls to prevent our anti-fraud from hard killing 3% of workloads, including many instances of pg"

Oof.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#318

Post by paganel »

Apparently this has nothing to do with real-world trains and to the real-world rail system, at first, and reading the title alone, I had thought that some trains might have got stuck somewhere because of an IT (google cloud) failure. It's just another SaaS story.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#319

Post by jkogara »

Interestingly, upon logging in this morning I was presented with a new terms and conditions banner that required me to agree to not deploy a list of, to varying degrees, nefarious things (bots, torrents, "anything illegal", etc.). Is it likely that some of these workloads resulted in the auto restriction from GCP?
Post reply on HN