Live data from Hacker News

Incident Report: May 19, 2026 – GCP Account Suspension

blog.railway.com

51–60 of 279 posts

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#51

Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month

I think the message here is that you can't trust any single cloud provider. You at least need two with full operational capability.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#52
post #50
post #33

The interesting and yet-to-be-explained part is why google flagged the account? Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause. The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.

That‘s the point where Google tells you they won’t tell you the exact reason because of security reasons

Exactly this, which is the problem with all modern accounts. No person to talk to so you can understand what happened and maybe fix it.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#53
post #42

I've read all the threads and their main page and I still don't really understand what this service is. Is this like a commercial alternative to Gerrit? What do people use this for? I'm not a developer, just curious what this is.

The category is “Platform as a Service”

Alternative to Fly or Heroku

Here is my source code Run it on the cloud for me I do not care how

In this case it looks like they also bundle together a bunch of the other services you would need to get code onto the platform, monitor it once it’s there and so on

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#54
post #24
post #12

I've been getting serious, recently, about moving all my workloads to equipment that I control in datacenters with which I have professional relationships. It's less expensive, easier, and this kind of nonsense doesn't happen. These cloud providers need to step back and observe how terrible they've made these products. Footguns everywhere, pricing that is impossible to forecast or reason about, broken APIs, and autom…

The thing that's nice about physical datacenters with people is that they often have to physically walk over to disconnect you - it's not as easy as some automated system doing an AI.

And if they do, you can walk over there too and ask a human why in person. (Or just call the NOC)

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#55
post #42

I've read all the threads and their main page and I still don't really understand what this service is. Is this like a commercial alternative to Gerrit? What do people use this for? I'm not a developer, just curious what this is.

The category is “Platform as a Service” Alternative to Fly or Heroku Here is my source code Run it on the cloud for me I do not care how In this case it looks like they also bundle together a bunch of the other services you would need to get code onto the platform, monitor it once it’s there and so on

Oh I see, so they manage the server hosting and application server configuration, optimization and all that jazz. Almost like one step away from managed hosting. Makes sense now, thankyou!

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#56
post #15

"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.

They have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#57

back to on-prem

Honestly, i have been wanting to suggest to my leaders that we should go to on-prem for primary, and use cloud only as extra for peak traffic and/or failover, etc...but, the culture where i'm at is so bought into cloud as if it solves all problems...and then, in the next breath they all ask me to drastically reduce cloud costs and ensure 100% uptime at all times 24/7/365 (1005 uptime without complexity and without any added costs!).

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#58

> May 19, 22:10 UTC - Our automated monitoring detected API health check failures and paged our on-calls, who started investigating the issue. > At 22:20 UTC on May 19, Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. If the timestamps are accurate, what was causing the errors 10 minutes before the account was suspended? The simplest explanation is…

That 10 minutes is likely very normal. Possibly...

* A Google employee messes up a setting (like one of the previous incidents) triggers something that looks like a suspension is warranted and it takes 10 minutes to flow through the process to suspend.

* A Railway customer does something corrupt, or seemingly corrupt, Google's system starts limiting access and take 10 minutes to decide it should be a suspension.

These are even more likely if there is a person in the loop to approve, who obvious did not dig deep enough to see that they should not have done so.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#60
post #15

"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.

Meta is no different. I know a company that had their OAuth app on Meta rendered completely unusable just because one of their employees (a dev) had their personal Facebook account banned by Meta for no reason. They tried to escalate it multiple times but got nowhere, lol. Meta is even worse because accounts need to be 'personal'; if you have a Business Manager, the users added to it are all tied to their personal Meta/Facebook accounts. This is ludicrous.
Post reply on HN