Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: May 19, 2026 – GCP Account Suspension

blog.railway.com

Post reply on HN
Print view View original

81–90 of 279 posts

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#81

Post by lotsoweiners »

malfist wrote: post #70
nthypes wrote: post #60

Earlier quoted context omitted.

Meta is no different. I know a company that had their OAuth app on Meta rendered completely unusable just because one of their employees (a dev) had their personal Facebook account banned by Meta for no reason. They tried to escalate it multiple times but got nowhere, lol. Meta is even worse because accounts need to be 'personal'; if you have a Business Manager, the users added to it are all tied to their personal Me…

[flagged]

Seems relevant to me as it is still a service that their company relied on.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#82

Post by jerf »

sa46 wrote: post #72
Animats wrote: post #15

"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.

Railway has an overwhelming incentive to pin the blame on Google. This report doesn't answer why Google suspended Railway's account. I'd wait for more details before adjudicating.

In principle, I agree with you.

In practice, Google has earned the way my priors are ready to believe it's 100% their fault with mighty and sustained effort. Or lack thereof, depending on your point of view.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#83

Post by daninsea »

Animats wrote: post #15

"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.

Railway don't have a great reputation for building scalable systems (effects of vibe coding?). It's worth waiting for Google's response before jumping to conclusions. They can move to Azure/AWS/own datacenter, but there's a good chance this will repeat in a few months.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#84

Post by gandreani »

QuercusMax wrote: post #18
Jgrubb wrote: post #2

Railway has not had the best month in the tech press have they? And in both cases it was an automated process belonging to some other party that put them there, damaging their reputation. I was going to talk to our google rep about their killing the Gemini cli but this is way more concerning.

Building on someone else's platform is always gonna be a risky move, and building a platform on top of someone else's platform is even riskier. My company used to use a hosting provider that was basically AWS plus some extra guarantees. We just finished migrating onto regular AWS because they now offer what we need directly.

But...AWS is a platform too, no? Seems like you're in the same category of risk you just moved to a more well-known name. Granted, Amazon is the most reliable even if they have their own quirks.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#85

Post by SoftTalker »

Aperocky wrote: post #44
tcdent wrote: post #33

The interesting and yet-to-be-explained part is why google flagged the account? Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause. The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.

Shouldn't Google answer this if they are unhappy with this incident report? Are we even sure that Railway knows?

The report at this point is pretty much just a timeline of what happened. No explanation of why, no accusations, no blame. A PR piece, to Railway's customers, reassuring them that "we're not ignoring this."

Now the lawyers are huddling. IMO there won't be a lot more said publicly by either side, at least until any threat of lawsuits for damages is settled.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#86

Post by xp84 »

levkk wrote: post #14
ryanSrich wrote: post #11

Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month

DigitalOcean. Seriously. They have been around a long long time and built a lot of the core infrastructure you rely on every day (e.g. Ceph).

I have read plenty of snark about them on HN, but I found their product incredibly useful, well-designed, and easy to work with. If I was building a new startup from scratch, I'd definitely be giving them a look.

I'm sure there are plenty of the like 1,000 AWS products that DO has no viable competitor for, but for what they do offer, they're great.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#87

Post by tamimio »

> Railway’s production account into a suspended status incorrectly, as part of an automated action.

Be it individuals or companies, this time is the best time to ditch all dependence on anything clouds or SaaS since all are using automated AI, more and more of these incidents will occur.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#88

Post by CPLX »

sa46 wrote: post #72
Animats wrote: post #15

"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.

Railway has an overwhelming incentive to pin the blame on Google. This report doesn't answer why Google suspended Railway's account. I'd wait for more details before adjudicating.

They said it was automated and affected a bunch of other customers, which gives at least some hint.

And in general Google lost any immediate benefit of the doubt status many years ago. Many such stories.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#89

Post by xp84 »

danjl wrote: post #51
ryanSrich wrote: post #11

Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month

I think the message here is that you can't trust any single cloud provider. You at least need two with full operational capability.

Yup. I don't know enough people at giant companies to know how many actually do this though. Not just talking having 2 AZs, I'm talking about ability in a DR scenario to fail over, within 5-10 minutes, to a different cloud provider, e.g. AWS → Hetzner, or GCP → Azure.

My gut feeling is that the number of significant applications that have this capability can probably be counted on two hands. Especially since a lot of the largest footprints of software stacks running in the cloud belong to Google and Microsoft, who I'm pretty sure do not replicate their services into someone else's cloud.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#90

Post by rescbr »

raverbashing wrote: post #38
rurban wrote: post #4

Google, the new Microsoft!

Amazon played AWS from day 1 as if they were the runner-up (and in a sense they were), and while it does look like it's day 2 there, they are not letting the momentum down Microsoft might have technical warts but commercially they are strong and Azure is a lot of times bundled with other services and you know you can get someone on the phone if needed Google has... ?

> Google has... ?

former Oracle salespeople

Post reply on HN