351–360 of 381 posts
>We have resolved this incident and a post mortem is available here. > https://blog.railway.com/p/incident-report-may-19-2026-gcp-a... >May 20, 07:57 UTC https://status.railway.com/incident/I23M92U0
It should be possible to sue Google for damages in such cases. This isnt a network outage or service failure which I would consider part of ToS.
Earlier quoted context omitted.
This is the conflict at the center of running a hosting company - make it easy to signup and you get a lot of new users but also a lot of abuse. Implement anti-abuse measures and you will hit some loud false positives (this may be the case with GCP here). I don't envy anybody running a hosting co - the internet is a really ugly place under the surface. edit: to add - AWS are really good here. Must be the ~30 years of…
Is it really a false positive if railway lets people run abusive services on GCP and then GCP consequently shuts them down?
seriously, is it possible to trust GCP with critical data/services at this point if you're not a billion dollar company? I'm exaggerating but someone said they got "auto banned" what if that happens to a small account which hosts some really important data/services there?
Railway isnt far from being a billion dollar company, no ?
Earlier quoted context omitted.
GCP has a lot of customers. But you wouldn't know the companies that do, unless you worked there and wanted to leak it, or it publicly comes out. Eg it's been publicly acknowledged that Apple uses GCP for iCloud, https://www.cnbc.com/amp/2018/02/26/apple-confirms-it-uses-g... , and Home Depot is another that's used as a case study, https://cloud.google.com/customers/the-home-depot but most customers don't want to mak…
upvoted & favourited because you taught me a really interesting fact which I feel makes up for an amazing discussion (regarding icloud using GCP). also, I can't help but imagine if instead of render, it was Apple's account which could've been auto-banned (Render is almost a billion dollar company or series-B, I am not sure) I haven't read the articles and I admit that but can you please elaborate to me on why Apple u…
I believe you mean Railway.
Render (a $1.5B company) has been hosting customers on GCP since 2018, and has never been banned.
Earlier quoted context omitted.
It should be possible to sue Google for damages in such cases. This isnt a network outage or service failure which I would consider part of ToS.
What if the reason for their stuff being shut down was a payment issue like an expired credit card or maxed credit account? Unless I missed it skim reading their post I don’t see any information anywhere about their communications with Google.
> May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly.
Earlier quoted context omitted.
> But TheRegister did reach out to Google and they have not replied yet That is exactly what GCP should do: not comment on a customer's issues. Even when it's due to abuse from a customer, which might even be the case.
Railway are alleging that this affected many GCP accounts, which would make at least a confirmation or denial of scope appropriate.
From what I've read in other comments, the root cause seems to have been automatic account suspension as an anti-abuse measures.
It's also telling that Railway describes the root cause simply as "Google Cloud Platform has suspended Railway's production account." It then mentions this
> At 22:20 UTC on May 19, Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
The why is conspicuously absent, but this sort of sweep is indeed consistent with anti-abuse measure.
If this is the case I would be cautious in accusing a cloud provider of wrongdoing. Many things need to go awfully wrong to trigger this sort of alarm, and I'm not talking about GCP's anti-abuse system. In fact, it's telling that no reputable, well established business is reporting any impact. The whole point of any anti-abuse system is to suspend accounts that are caught engaging in some sort of abuse.
Earlier quoted context omitted.
It should be possible to sue Google for damages in such cases. This isnt a network outage or service failure which I would consider part of ToS.
It's always possible to sue, but Google has good terms of service and lawyers - I'm 99% confident that a lawsuit would end up nowhere.
But that's not what will happen. Google will offer an apology (perhaps even a public one), a giant pile of account credit, and a pinky promise not to do it again. Railway will accept it and hmmm and haw internally about whether to decrease their reliance on GCP, and then when they calculate the cost of going in on other clouds more heavily (or their own metal), they'll just think harder about weird failure modes.
Earlier quoted context omitted.
"we did not have the monitoring or controls to prevent our anti-fraud from hard killing 3% of workloads, including many instances of pg" Oof.
Needs an anti-anti-fraud service which terminates malfunctioning anti-fraud services.
Earlier quoted context omitted.
There would have been efforts to contact them, but it would have been via their contact method, aka the email they set it up with. Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation. Sign up for terms. It will get you payment terms!
Honestly still insane to nuke a high-volume client's business after a single payment issue. There would be no reason for Google to believe that a single hiccup like that is evidence that they won't get paid and have to cut account access immediately.
I'd be curious to know why Railway's account was suspended. Was it a similar payment issue or something else?