Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

321–330 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#321

Post by theobr »

tardwrangler wrote: post #248

Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.

I got details I shouldn't have. I can confidently say this one's 100% on Google, and I will be disappointed if Railway is unable to share more. There is literally nothing they could have done to prevent this aside from avoiding GCP entirely.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#322

Post by captn3m0 »

darkwater wrote: post #313
realityking wrote: post #290

Earlier quoted context omitted.

My experience with a large-ish ($5m/year) AWS account was quite different. They were happy to support us with cost optimizations, discounts, and one time credits for certain activities (co-innovation and archiving certain milestones in their partner program). Their primary concern seemed to have been to keep as much of our workload inside AWS as possible and to win workload from 3rd party services we used (e.g. CDNs)…

This is my experience as well with AWS accounts in the 2-4M/year range, the biggest upsell they always try to make is Enterprise Support, but for the rest they are usually happy to help you cut cost in the short term - as long as you stay with them in the long term.

The incentives are nicer here, from what I’ve heard: AWS TAMs are not reviewed in revenue at all. And cost savings for customers actually counts as a win for them.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#323

Post by tardedmeme »

rajeshvar wrote: post #187
guluarte wrote: post #109

Earlier quoted context omitted.

It's Google. They let you use their services, but the moment you don't fit the norm, they suspend you.

What does blocked mean? Is there a different post that I am missing? There is shared infrastructure in GCP for networking (ex-googler here) and if only railway is affected, then it is not clear if it is only GCP or if there is something from Railway's perspective that needs to be addressed.

It means they ban you. Cancel your account. No takesie backsies. Say bye bye to your revenue. This can happen at any time for any reason or no reason and it's amazing nobody's learned from all the other times it happened.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#324

Post by sschueller »

r721 wrote: post #312

>We have resolved this incident and a post mortem is available here. > https://blog.railway.com/p/incident-report-may-19-2026-gcp-a... >May 20, 07:57 UTC https://status.railway.com/incident/I23M92U0

It should be possible to sue Google for damages in such cases. This isnt a network outage or service failure which I would consider part of ToS.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#325

Post by raverbashing »

kevin_nisbet wrote: post #89
tjpnz wrote: post #59

Earlier quoted context omitted.

AWS normally contacts you first.

Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

If a vendor doesn't know the basics about pentesting open infra and can't be bothered to look up terms of use sounds like they know ssh-it about fsck

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#326

Post by quentindanjou »

r721 wrote: post #312

>We have resolved this incident and a post mortem is available here. > https://blog.railway.com/p/incident-report-may-19-2026-gcp-a... >May 20, 07:57 UTC https://status.railway.com/incident/I23M92U0

Railway say the incident is resolved but many are still down (returning 502): on our side, we had to manually trigger a redeploy to fix it but I believe it should have been triggered automatically by Railway and I can't understand how they can mark this as resolved while many are still down.

In total, down for >11 hours on our side.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#328

Post by whh »

macintux wrote: post #88
MrDarcy wrote: post #77

Earlier quoted context omitted.

There’s no way this was automated or silent. The only reasonable explanation is Railway lost control of their estate and something was happening that warranted a group of humans to decide flipping the kill switch was the best of a set of bad alternatives.

You’re giving Google far more credit than they’ve earned.

It's almost certainly one of those Android Store related checks or YouTube account checks. It's why it's best to disable login for the services you don't want your staff messing with on Google Workspace.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#329

Post by nikcub »

swyx wrote: post #292
nikcub wrote: post #178

Earlier quoted context omitted.

This is the conflict at the center of running a hosting company - make it easy to signup and you get a lot of new users but also a lot of abuse. Implement anti-abuse measures and you will hit some loud false positives (this may be the case with GCP here). I don't envy anybody running a hosting co - the internet is a really ugly place under the surface. edit: to add - AWS are really good here. Must be the ~30 years of…

i wonder if DID or World (various ways of Proof of Human) can help solve this issue.

This just incentivizes market for bio-mules, which already exists with world[0] - where prices stay low because it was rolled out to low-income countries.

Then there's the platform game theory. If you adopt you add friction which reduces signups, and there will always be a competitor who would risk the 10x fraud increase in order to capture 100x the market. Railway has seen hyper-growth because it's so easy to run from, and is recommended by, coding agents[1].

The solutions are here already just not well implemented or understood - probabilistic fraud detection, resource limits, service and automation limits, standard gov identity verification as a signal, enterprise sales channels with human relationships, etc.

There are tradeoffs with each platform choice that just aren't well understood. Most users shop on price and DX and don't see the abuse infra or problem until it hits them.

Google and GCP have a problem where they completely cook users who get flagged in their automated fraud net (this isn't news - or shouldn't be)

[0] https://www.coindesk.com/policy/2023/05/24/black-market-for-...

[1] and the problems that come with providing that simple interface, like sometimes dropping prod

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#330

Post by locknitpicker »

gizzlon wrote: post #276
locknitpicker wrote: post #263

Earlier quoted context omitted.

> Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. This. It's very odd that in other threads we see a bunch of accounts heavily invested in criticizing a cloud provider, but what's conspicuously absent from this wave of indignation is any curiosity in the root cause, or even any interest in exploring what it might have been. Quite odd.

Agreed, I'm very curious as to how this could happen. But TheRegister did reach out to Google and they have not replied yet: https://www.theregister.com/off-prem/2026/05/20/google-cloud...

> But TheRegister did reach out to Google and they have not replied yet

That is exactly what GCP should do: not comment on a customer's issues. Even when it's due to abuse from a customer, which might even be the case.

Post reply on HN