"Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it." - Thanks Claude!
Incident Report: May 19, 2026 – GCP Account Suspension
91–100 of 279 posts
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#92Earlier quoted context omitted.
I;ve had my share of VPS & Managed DB outages at DO, so they are also not faultless.
I've been with DO since checks mailbox 2014. Honestly never experienced an unannounced outage.
Yeah overall they are ok. I think 3 times managed db and one or twice a vps just dead. No issues in a year or so.
They were always hardware failures, took about 45-120min. Not the end of the world, but also not fun getting lot of client complaints.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#93This should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions. TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work.…
And this is Railway, a big enough name to top the HN main page and presumably find someone from Google to intervene at some point. I would have zero recourse if it was some little product that I built.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#94They forgot to get reimbursement for downtime. A free month of GCP is better than nothing.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#95Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month
An intermediary can provide value but there’s also a risk so I’d consider why you don’t want to use AWS, GCP, etc. directly. All of the major cloud providers have services which are only slightly harder than what Railway does but allow you to grow into more advanced things as your needs expand without adding a third-party who controls your features, security, and availability.
As an example, I note that GCP responded within 7 minutes according to their timeline. If you’d been using Cloud Run, that would have reduced downtime by over 7 hours — and there’s a good chance that you never would have gone down in the first place if the unknown trigger event was related to other customer activity or something odd Railway did.
There’s also a complexity factor: note how much complex infrastructure they mentioned having to fix that you wouldn’t need for your own account. That code does useful things, I’m sure, but it’s also a lot of moving parts which a hosting provider needs and you don’t – this outage took everyone down, whereas individual AWS or bare metal users would’ve otherwise been unaffected. There isn’t a global optimum which is the same for everyone but I think developers are prone to wildly over-estimating how much time they save by removing a couple of deployment steps relative to the direct costs and the less obvious costs of working within someone else’s environment.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#96The interesting and yet-to-be-explained part is why google flagged the account? Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause. The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.
I don't think you're typically told why for these things, and it's mostly automated from what I can tell. The automated systems make mistakes but more importantly they're completely opaque. Nobody, not even Google, knows how they work exactly.
Google should know why a human accepted the automated suggestion, or if and why there wasn't any human oversight in the first place.
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#97"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.
They have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.
The cloud provider in question - GCP - who also deleted a 125 billion dollar company's entire account on accident?
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#98Unfortunately we had to make emergency migration off to Azure yesterday due to this. Thankfully our DB was not hosted on Railway and we were back up in a couple hours. As much as we loved the simplicity they provided us, there's just been too many mishaps and shortcomings for us to continue running a B2B enterprise app on their infrastructure. Sad day :(
Azure suspended your account as well?
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#99"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.
Meta is no different. I know a company that had their OAuth app on Meta rendered completely unusable just because one of their employees (a dev) had their personal Facebook account banned by Meta for no reason. They tried to escalate it multiple times but got nowhere, lol. Meta is even worse because accounts need to be 'personal'; if you have a Business Manager, the users added to it are all tied to their personal Me…
Yeah, people loose their business because a kid is logged in on their iPad, gets their google account suspended, and google knows it's the same household as the parent, and everything gets shut down
Re: Incident Report: May 19, 2026 – GCP Account Suspension
#100Earlier quoted context omitted.
Building on someone else's platform is always gonna be a risky move, and building a platform on top of someone else's platform is even riskier. My company used to use a hosting provider that was basically AWS plus some extra guarantees. We just finished migrating onto regular AWS because they now offer what we need directly.
But...AWS is a platform too, no? Seems like you're in the same category of risk you just moved to a more well-known name. Granted, Amazon is the most reliable even if they have their own quirks.
Each critical dependency you stack multiplies your risk. Now you have to worry about Railway AND Google causing business-damaging outages.