Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: May 19, 2026 – GCP Account Suspension

blog.railway.com

Post reply on HN
Print view View original

201–210 of 279 posts

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#201

Post by christophilus »

acdha wrote: post #95
ryanSrich wrote: post #11

Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month

An intermediary can provide value but there’s also a risk so I’d consider why you don’t want to use AWS, GCP, etc. directly. All of the major cloud providers have services which are only slightly harder than what Railway does but allow you to grow into more advanced things as your needs expand without adding a third-party who controls your features, security, and availability. As an example, I note that GCP responded…

This entire thread illustrates why you don’t want Google in any critical part of your business. AWS, sure. Azure? Maybe. I’m not familiar with Azure, but if I have to pick one, it’s AWS.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#204

Post by jiggawatts »

xp84 wrote: post #89
danjl wrote: post #51

Earlier quoted context omitted.

I think the message here is that you can't trust any single cloud provider. You at least need two with full operational capability.

Yup. I don't know enough people at giant companies to know how many actually do this though. Not just talking having 2 AZs, I'm talking about ability in a DR scenario to fail over, within 5-10 minutes, to a different cloud provider, e.g. AWS → Hetzner, or GCP → Azure. My gut feeling is that the number of significant applications that have this capability can probably be counted on two hands. Especially since a lot of…

Before the cloud it was commonplace to have redundant data centres from two or more colocation provider companies. Similarly, Internet uplink diversity was commonplace.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#205

Post by adrr »

shwetanshu21 wrote: post #47

This should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions. TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work.…

GCP was never known for their support and deprecation of services was always a huge risk. Its very sad because its actually a quality product. They should easily be the number 2 provider. Azure is extremely unreliable and their documentation is subpar. GCP being in 3rd place is more of their doing.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#206

Post by 0cf8612b2e1e »

r0m4n0 wrote: post #166
Animats wrote: post #15

"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.

I'm not sure that's the lesson to learn from this outage. Hell Google resolved the problem in 7 minutes which is as good as you could hope for. The resulting action should be you have proper disaster recovery, failover, etc. Not sure I would trust these folks if this is the conclusion they are coming to from this experience. Any cloud provider can/will do this to you.

Google resolved the problem in seven minutes for a billion dollar company. Good luck if you are a nobody.

The best you could hope for is that if there is something fishy with your account, you are contacted by Google to address it.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#207

Post by stackskipton »

londons_explore wrote: post #119
JCTheDenthog wrote: post #77

Earlier quoted context omitted.

Unfortunately the cloud providers also rarely if ever tell you the reason.

My guess would be the credit card expired.... If it were something out of Railways hands, I think they would say something like "We have not yet identified the reason for the suspension, and are awaiting a response from Google".

At any company doing Enterprise work, you don't cut off someone for non payment without Account Manager doing multiple phone calls to whoever you have contact information for, emailing everyone listed on the account and whoever opened a support ticket and maybe even putting a banner in the panel with "ACCOUNT OVERDUE, CALL US TO SORT IT OUT!"

Generally it takes 30 days past due and complete no contact for anyone before suspension.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#208

Post by mjcl »

llmslave wrote: post #134
array_key_first wrote: post #66

Earlier quoted context omitted.

I don't think you're typically told why for these things, and it's mostly automated from what I can tell. The automated systems make mistakes but more importantly they're completely opaque. Nobody, not even Google, knows how they work exactly.

For big accounts, like railway, zero chance this was a handsoff fully automated ban

Really? This isn't the first time their automation took down a big customer (UniSuper in 2024) by accident. In that case the automation actually deleted the resources and GCP had to recover them.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#209

Post by 8note »

adrr wrote: post #205
shwetanshu21 wrote: post #47

This should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions. TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work.…

GCP was never known for their support and deprecation of services was always a huge risk. Its very sad because its actually a quality product. They should easily be the number 2 provider. Azure is extremely unreliable and their documentation is subpar. GCP being in 3rd place is more of their doing.

i wouldnt really call it a product without support, let alone a good product.

its a nicely design hobby, that somebody could make a good product out of, by following the same abstractions

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#210

Post by 8note »

shwetanshu21 wrote: post #154

"Railway owns our vendor choices, and we ultimately own this one. Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it." Kudos to them for acknowledging it and not doing PR speak. It shows it was an architectural failure from their part of trusting GCP, and they are working to fix it. Should they have seen it com…

it sounds to me like templated text from the UVic ESS office somewhere:P
Post reply on HN