Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: May 19, 2026 – GCP Account Suspension

blog.railway.com

Post reply on HN
Print view View original

31–40 of 279 posts

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#31

Post by bink »

sroussey wrote: post #19
Shank wrote: post #17

Earlier quoted context omitted.

> If the timestamps are accurate, what was causing the errors 10 minutes before the account was suspended? Assuming the timestamps are accurate, Google probably started terminating resources while the account was not "suspended" and only completed that after all resources were disabled.

Or the account started doing something nefarious (assuming one of their customers as root cause, not railway itself) that started causing real problems and Google shut it down. The problem with not having the data is that it’s easy to make assumptions.

The absence of any explanation for the suspension does seem intentional. If it were me that's one of the first things I would've asked so that I could make sure it doesn't happen again.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#32

Post by SoftTalker »

stefan_ wrote: post #10

It's reassuring to know they will ban a million dollar enterprise customer just like they will ban your GMail of 20 years.

It really is amazing that there is not some level at which "human review" becomes mandatory. Customers of that size already have dedicated account rep contacts.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#33

Post by tcdent »

The interesting and yet-to-be-explained part is why google flagged the account?

Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause.

The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#34

Post by majdalsado »

Unfortunately we had to make emergency migration off to Azure yesterday due to this. Thankfully our DB was not hosted on Railway and we were back up in a couple hours.

As much as we loved the simplicity they provided us, there's just been too many mishaps and shortcomings for us to continue running a B2B enterprise app on their infrastructure.

Sad day :(

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#35

Post by wouldbecouldbe »

levkk wrote: post #14
ryanSrich wrote: post #11

Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month

DigitalOcean. Seriously. They have been around a long long time and built a lot of the core infrastructure you rely on every day (e.g. Ceph).

I;ve had my share of VPS & Managed DB outages at DO, so they are also not faultless.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#36

Post by Illniyar »

ryanSrich wrote: post #11

Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month

If you are unable to use IaaS directly. You need to accept that your service might be down.

Even if you use AWS and the like, if you aren't building your app with redundancy across multiple AZs, then you'll have some downtime occasionally.

And even if you do build redundancy with multiple AZ, some services might fail anyway as AWS is not entirely isolated. So you might have downtimes.

So just accept downtimes and use the best tool for you (unless they are really bad, like GitHub level bad). If you cannot accept any downtime, you'll have to spend millions of dollars and months of work to have the confidence to expect no downtime. Something like Netflix's chaos monkey and infrastructure would be enough.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#37

Post by jasonlotito »

wouldbecouldbe wrote: post #35
levkk wrote: post #14

Earlier quoted context omitted.

DigitalOcean. Seriously. They have been around a long long time and built a lot of the core infrastructure you rely on every day (e.g. Ceph).

I;ve had my share of VPS & Managed DB outages at DO, so they are also not faultless.

Not if, but when. No one is faultless. Chasing after 100% is a fool's errand.

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#38

Post by raverbashing »

rurban wrote: post #4

Google, the new Microsoft!

Amazon played AWS from day 1 as if they were the runner-up (and in a sense they were), and while it does look like it's day 2 there, they are not letting the momentum down

Microsoft might have technical warts but commercially they are strong and Azure is a lot of times bundled with other services and you know you can get someone on the phone if needed

Google has... ?

Re: Incident Report: May 19, 2026 – GCP Account Suspension

#39

Post by tedd4u »

stefan_ wrote: post #10

It's reassuring to know they will ban a million dollar enterprise customer just like they will ban your GMail of 20 years.

I can't believe Kurian has not put his foot down about this. Adverse action against accounts over $X ARR absolutely must have review by revenue-carrying people before the action is taken.
Post reply on HN