TL;DR: putting all your eggs into one basket is bad, man.
Note, you submitted a dupe: https://news.ycombinator.com/item?id=48201711 (the comment I'm replying to is 1 ID older so I guess this is the canonical one to reply to)
Incident Report: Railway Blocked by Google Cloud [resolved]
361–370 of 381 posts
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#362>We have resolved this incident and a post mortem is available here. > https://blog.railway.com/p/incident-report-may-19-2026-gcp-a... >May 20, 07:57 UTC https://status.railway.com/incident/I23M92U0
Thanks! The post-mortem is currently on the frontpage here:
Incident Report: May 19, 2026 – GCP Account Suspension - https://news.ycombinator.com/item?id=48204770
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#363Earlier quoted context omitted.
upvoted & favourited because you taught me a really interesting fact which I feel makes up for an amazing discussion (regarding icloud using GCP). also, I can't help but imagine if instead of render, it was Apple's account which could've been auto-banned (Render is almost a billion dollar company or series-B, I am not sure) I haven't read the articles and I admit that but can you please elaborate to me on why Apple u…
> also, I can't help but imagine if instead of render, it was Apple's account which could've been auto-banned (Render is almost a billion dollar company or series-B, I am not sure) I believe you mean Railway. Render (a $1.5B company) has been hosting customers on GCP since 2018, and has never been banned.
Yes sorry I meant railway rather than render so sorry about that, it was more of an honest mistake!
> Render (a $1.5B company) has been hosting customers on GCP since 2018, and has never been banned.
speaking of which, I have a question for render but how does render prevent something like what has happened with railway (ie. the account getting banned), I would love to know more about what the team at render thinks of such and also I would love to get some thoughts on why Render is using GCP, I would love to know some architectural decisions behind it as I am curious about it!
Once again, thanks for responding to me and waiting for your response and have a nice day anurag!
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#364Earlier quoted context omitted.
Isn't that including things like google workspace and similar? Both Azure and GCP have sometimes included things that most people think of as unrelated SaaS (office 365, gsuite/workspace) to make themselves look bigger in the cloud sector.
> Isn't that including things like google workspace and similar? AWS also includes Amazon WorkSpaces. Moreover, AWS includes all of Amazon's cloud infrastructure for things like Amazon music, Ring, Amazon Prime Video, etc.
But as a percentage of revenue I'd assume those are a lot smaller than Office365 is for microsoft and Workspace is for google.
Last I checked I don't think AWS included things like Amazon Prime Video either, AWS is primarily their buissness/platform offerings, not consumer things like Twitch/Prime/Music/etc.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#365Earlier quoted context omitted.
It's always possible to sue, but Google has good terms of service and lawyers - I'm 99% confident that a lawsuit would end up nowhere.
They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…
One company I worked at is highly reliant on Google Cloud, but at one point we moved some services to Azure.
Azure noticed, and immediately hit us with a discount offer in the hopes of getting more of our business.
Google noticed, and immediately hit us with a discount offer in the hopes of keeping more of our business.
This is just a reminder that your multi-cloud strategy doesn't have to be 'deploy everything across multiple clouds'; it can even just be 'make it obvious that you have leverage'.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#366Earlier quoted context omitted.
It's always possible to sue, but Google has good terms of service and lawyers - I'm 99% confident that a lawsuit would end up nowhere.
They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…
I'm sure their contract explicitly states that their account can be suspended or terminated immediately without prior notice upon violating some TOS. And, most TOS are incredibly wide and vague, it wouldn't necessarily be hard to find something they violated.
This is sort of the problem with these new-age internet companies. The contracts are incredibly hostile. Most TOS you see amount to "you have no rights and we can fuck you up the ass"
Google is a B2C company so I'm sure some of that culture transfers over to B2B relations, but I'm speculating. Maybe the contracts are more normal for B2B.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#367Earlier quoted context omitted.
They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…
I'm sure their contract explicitly states that their account can be suspended or terminated immediately without prior notice upon violating some TOS. And, most TOS are incredibly wide and vague, it wouldn't necessarily be hard to find something they violated. This is sort of the problem with these new-age internet companies. The contracts are incredibly hostile. Most TOS you see amount to "you have no rights and we c…
[dead]
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#368Earlier quoted context omitted.
There would have been efforts to contact them, but it would have been via their contact method, aka the email they set it up with. Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation. Sign up for terms. It will get you payment terms!
Honestly still insane to nuke a high-volume client's business after a single payment issue. There would be no reason for Google to believe that a single hiccup like that is evidence that they won't get paid and have to cut account access immediately.
It's not a single payment failure, it would be multiple days, possibly even a week to 10days.
This is why businesses should put in the effort and sign up for credit terms. Then it's an invoice, and you reduce this risk substantially.
Credit cards are _not_ reliable at this scale. Banks are offline all the time, cards are marked stolen, protocols change, all sorts of things that will cause flags indicating "the money can't move down that path".
Businesses that pay for AWS/Cloud/etc via credit cards are trying to buy reliability but put it behind a single point of failure.
Credit cards are not how you should be paying for business services with uptime requirements!
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#369Earlier quoted context omitted.
They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…
I'm sure their contract explicitly states that their account can be suspended or terminated immediately without prior notice upon violating some TOS. And, most TOS are incredibly wide and vague, it wouldn't necessarily be hard to find something they violated. This is sort of the problem with these new-age internet companies. The contracts are incredibly hostile. Most TOS you see amount to "you have no rights and we c…
If Railway's account was suspended due to an error, not a TOS violation, I doubt they'll pull such a card. If Google were sued, such a blatant lie would be found in discovery pretty quickly and I doubt a court would look upon it favourably. They'd also be starting a public PR battle that they'd almost certainly lose (assuming there was indeed no real TOS violation) which would make them look completely untrustworthy. So I doubt Google will do that.
bastawhiz is probably right in that Google will offer some credits and an apology, and Railway will reduce their dependence (probably), rather than a lawsuit. And I doubt Railway wants to be on the bad side of one of the few big cloud providers. But I'd be surprised if Railway didn't have a good argument to make for compensation or a lawsuit.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#370Earlier quoted context omitted.
I still remember the one where they nuked all the storage of I think an Australian insurance company I think, luckily the it department had done a multi cloud setup for backups
Google Cloud accidentally deletes $125 billion Australian pension fund - May 2024 https://www.business-standard.com/world-news/google-cloud-ac...