Now given the logic that you can't be dependent on any one service to run your SaaS, how does Railway convince its customers to run their SaaS on a single service?
Incident Report: May 19, 2026 – GCP Account Suspension
111–120 of 279 posts
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#112What drives Google to apply these actions so completely and immediately, versus a more deliberate approach, with notification and delay before action, manual review for paying customers, or a warning to resolve within X hours/days? Once or twice could be errors or bad implementation, but these can't explain away the pattern. It would seem that Google's counsel has deemed that whenever _____ is detected, the company m…
It could be automated action based on abuse reports. TONS of spam comes from Railway associated networks.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#113Earlier quoted context omitted.
Meta is no different. I know a company that had their OAuth app on Meta rendered completely unusable just because one of their employees (a dev) had their personal Facebook account banned by Meta for no reason. They tried to escalate it multiple times but got nowhere, lol. Meta is even worse because accounts need to be 'personal'; if you have a Business Manager, the users added to it are all tied to their personal Me…
Yeah, people loose their business because a kid is logged in on their iPad, gets their google account suspended, and google knows it's the same household as the parent, and everything gets shut down
> google knows it's the same household as the parent,
Nearly all these linkages are due to people sharing recovery email addresses and phone numbers. Don't do that.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#114> As a side effect, Terms-of-service acceptance records were also reset, prompting users to re-accept on their next visit to the dashboard.
Don't get me wrong- the rest of this mess falls pretty clearly on Google Cloud, but this one feels like something Railway did to themselves.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#115The interesting and yet-to-be-explained part is why google flagged the account? Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause. The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.
This exact thing happened to me when I ran https://www.fatherly.com/ circa 2017. Google just shut down our account without notice. We were spending like $10k/month. It also locked us out of our premium support account, so we couldn't even get anyone there to notice that they'd locked us out. After about 8 hours, a random Google support tech said it was because we were mining bitcoin, which was laughably untrue. We ha…
Google thinks everything should be replaced with automation.
Remember knowledge cards? Prior to the LLM AI revolution, they had an extraordinarily crappy AI system digest the entire internet to figure out the wrong facts about stuff and then present it to users as solid truth, with no human review and no way to report inaccuracies.
They just don't care. If the task requires a person to look at a thing and tell if it's right, they only do that for like 5 examples and then train a classifier, then deploy said classifier without thinking twice because "at internet scale" or whatever crap.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#116"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.
They have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.
LOL, did you woke up from the hibernation?
This is Google we're talking about. This absolutely happened many times in the past and will happen again.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#117Re: Incident Report: May 19, 2026 – GCP Account Suspension
#118"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.
They have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.
FTA:
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
This might be 100% of what google told them.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#119Earlier quoted context omitted.
They have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.
Unfortunately the cloud providers also rarely if ever tell you the reason.
My guess would be the credit card expired....
If it were something out of Railways hands, I think they would say something like "We have not yet identified the reason for the suspension, and are awaiting a response from Google".
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#120Railway has not had the best month in the tech press have they? And in both cases it was an automated process belonging to some other party that put them there, damaging their reputation. I was going to talk to our google rep about their killing the Gemini cli but this is way more concerning.
In the case of them giving AI admin credentials to delete their production database, and it deleted their production database: that's on them. They were the only ones who put the admin account credentials into their AI.
Then they took no personal responsibility. That definitely damaged their reputation. Here, they are taking at least some responsibility. Props to them on improving.
Also, GCP does indeed have serious reliability issues, and Google does indeed have serious customer support issues.
EDIT: It has been brought to my attention below that the first 2 paragraphs are misattributed, and were not Railway, but rather a customer of theirs. Sorry, Railway!