Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: May 19, 2026 – GCP Account Suspension

blog.railway.com

Post reply on HN
Print view View original

251–260 of 279 posts

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#251

Post by scott_w »

neya wrote: post #247
cube00 wrote: post #243

Earlier quoted context omitted.

> (What did the account manager say exactly?) I doubt we'll ever know, especially if it makes Google look negligent (ie. not reaching out to the customer first before restricting their production account) Whatever the account manager said didn't inspire confidence that this wouldn't happen again. Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for seco…

> Whatever the account manager said didn't inspire confidence that this wouldn't happen again. Or..they couldn't remain compliant with one of the strictest cloud vendors' policies.

Yet their account was restored so clearly GCP themselves disagree with you.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#252

Post by neya »

scott_w wrote: post #251
neya wrote: post #247

Earlier quoted context omitted.

> Whatever the account manager said didn't inspire confidence that this wouldn't happen again. Or..they couldn't remain compliant with one of the strictest cloud vendors' policies.

Yet their account was restored so clearly GCP themselves disagree with you.

Or they fixed the issue to remain compliant so GCP would restore access. Again, I know it's fashionable to hate on Google here, but there are always 2 sides to every story.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#253

Post by Shank »

neya wrote: post #229
shwetanshu21 wrote: post #47

This should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions. TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work.…

This is just typical anti-Google FUD. Been on GCP for nearly a decade and as do my peers. Sure, you hear about a few stories like these and in this particular case with Railway, I would actually wait to see what caused the trigger for the suspension - both from Railway and from GCP. But, this sort of thing happens with every cloud provider including AWS (you can google for the same thing "AWS shut down our account wi…

> As a former GCP consultant, I can share that these sort of shut downs aren't random and it's usually due to the customer not being compliant - that breaks cloud compliance requirements for the big clouds, so automated systems flag it. Eg. Someone serving CP on their CDN, for instance.

If this was the case it would obviously be horrific. I did check to see, and I noticed that Railway is not listed as an ESP who sent any reports to NCMEC / CyberTipline in 2025, which seems like the wrong number for a hosting provider. Maybe they just have absurdly good customers.

https://www.missingkids.org/content/dam/missingkids/pdfs/202...

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#254

Post by haute_cuisine »

ryanSrich wrote: post #11

Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month

Render has been solid so far

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#255

Post by shwetanshu21 »

keeda wrote: post #227
shwetanshu21 wrote: post #47

This should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions. TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work.…

At least from the outside TK seems to be doing well given GCP's growth. My completely uninformed assessment is that he stepped in as the disciplined adult in the room to override Google's otherwise lackadaisical approach to enterprise. (Clearly still some ways to go, as this incident shows.) Now, that may have created a culture that is at odds with the rest of Google, but it was probably required to become a "serious…

I think GCP has been giving out lucrative contracts with lots of discounts and credits which has led to this growth. I've seen many companies jump the ship cause they got 100/200k 2/3 year credits which is a lot for many of them at that stage. They also have their partners migrate them at their own expense. So it is essentially free money for those companies. I know a prominent entertainment content provider who went from AWS to Azure and then from Azure to GCP just because of this in 6-7 year. Plus the GPU hunt nowadays along with everyone producing code has definetely helped every cloud provider and even dedicated server providers/VPS (most of them dont have cheap options availaible). So how much of the growth has been organic vs TK led, hard to know.

If I think about it you might be right though some of this comes from Google's disdain for its customers be it hardware/software/cloud. So it is kinda inherited. One of the main reasons most good cloud consultants ask their clients to stay away from GCP is because of their depreciation policy. Plus stuff like this has been common in GCP.

So TK brought it to fix it instead of fixing it has brought the toxic OCI culture too which in the end brings them down even more.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#256

Post by scott_w »

neya wrote: post #252
scott_w wrote: post #251

Earlier quoted context omitted.

Yet their account was restored so clearly GCP themselves disagree with you.

Or they fixed the issue to remain compliant so GCP would restore access. Again, I know it's fashionable to hate on Google here, but there are always 2 sides to every story.

That’s not what you just claimed. I’ve quoted you below:

> Or..they couldn't remain compliant with one of the strictest cloud vendors' policies.

Which is it? They can comply or they can’t? Both sides can’t be true at the same time.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#258

Post by Hikikomori »

boylan wrote: post #105
tcdent wrote: post #33

The interesting and yet-to-be-explained part is why google flagged the account? Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause. The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.

This exact thing happened to me when I ran https://www.fatherly.com/ circa 2017. Google just shut down our account without notice. We were spending like $10k/month. It also locked us out of our premium support account, so we couldn't even get anyone there to notice that they'd locked us out. After about 8 hours, a random Google support tech said it was because we were mining bitcoin, which was laughably untrue. We ha…

Would AWS even care? They didn't when one obs guy was mining shitcoins with unused cpu time. Our security team did care.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#259

Post by skarz »

boylan wrote: post #105
tcdent wrote: post #33

The interesting and yet-to-be-explained part is why google flagged the account? Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause. The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.

This exact thing happened to me when I ran https://www.fatherly.com/ circa 2017. Google just shut down our account without notice. We were spending like $10k/month. It also locked us out of our premium support account, so we couldn't even get anyone there to notice that they'd locked us out. After about 8 hours, a random Google support tech said it was because we were mining bitcoin, which was laughably untrue. We ha…

$10k a month on a blog??

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#260

Post by pocksuppet »

stingraycharles wrote: post #235
okanat wrote: post #185

Earlier quoted context omitted.

Google knows and wants that there is no oversight. Don't do business with any big tech, if you don't want this kind of incidents.

AWS and Microsoft don’t do this, not like this.

AWS does it, but only for small customers, not for big customers without a human review.

Azure is just so broken they don't have to suspend your account to cause an outage.

Post reply on HN