Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

371–380 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#371

Post by anurag »

Imustaskforhelp wrote: post #363
anurag wrote: post #354

Earlier quoted context omitted.

> also, I can't help but imagine if instead of render, it was Apple's account which could've been auto-banned (Render is almost a billion dollar company or series-B, I am not sure) I believe you mean Railway. Render (a $1.5B company) has been hosting customers on GCP since 2018, and has never been banned.

Yes sorry I meant railway rather than render so sorry about that, it was more of an honest mistake! > Render (a $1.5B company) has been hosting customers on GCP since 2018, and has never been banned. speaking of which, I have a question for render but how does render prevent something like what has happened with railway (ie. the account getting banned), I would love to know more about what the team at render thinks o…

Without knowing more about why Railway was banned, it's hard to say how we would prevent it. Render uses GCP, AWS, and now, our own hardware [1]; GCP is mostly a historical artifact; we started using AWS for most things after dealing with poor customer and sales support from GCP a few years ago.

[1] https://x.com/anuraggoel/status/2057245946652901809

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#372

Post by Ferret7446 »

bastawhiz wrote: post #358
Cthulhu_ wrote: post #350

Earlier quoted context omitted.

It's always possible to sue, but Google has good terms of service and lawyers - I'm 99% confident that a lawsuit would end up nowhere.

They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…

I don't think you understand the purpose of the courts. They are not the first resort, they are the last resort, after you have tried all other (non violent) methods to resolve any grievances with another party.

I'm sure there are/will be discussions privately between the two to figure out a resolution.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#373

Post by nikanj »

bastawhiz wrote: post #358
Cthulhu_ wrote: post #350

Earlier quoted context omitted.

It's always possible to sue, but Google has good terms of service and lawyers - I'm 99% confident that a lawsuit would end up nowhere.

They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…

They have no right to sue, because their contract includes mandatory binding arbitration clauses.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#376

Post by daohieu91 »

Small Railway customer here ($5-10/mo, Spring Boot service). My service stayed up the whole time on May 19 — I only learned about the incident from this post-mortem. "Detailed post-mortem about an incident the customer didn't even notice" is the strongest trust signal Railway could have given.

The broader concern though isn't Railway-specific: GCP's "we can suspend a downstream provider with no warning" policy is a structural risk for every IaaS-on-IaaS layer (Fly, Render, Northflank, etc.). Curious whether anyone here has seen contractual or technical mitigations beyond "have a second deploy target ready to switch to."

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#377

Post by bastawhiz »

Ferret7446 wrote: post #372
bastawhiz wrote: post #358

Earlier quoted context omitted.

They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…

I don't think you understand the purpose of the courts. They are not the first resort, they are the last resort, after you have tried all other (non violent) methods to resolve any grievances with another party. I'm sure there are/will be discussions privately between the two to figure out a resolution.

You clearly didn't read my second paragraph, which starts "But that's not what will happen..."

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#378

Post by bastawhiz »

nikanj wrote: post #373
bastawhiz wrote: post #358

Earlier quoted context omitted.

They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…

They have no right to sue, because their contract includes mandatory binding arbitration clauses.

Absolutely not.

1. This isn't self service GCP, this is an enterprise contract. You don't build your medium sized business on a provider without one. Once you're a certain size, you don't get binding arbitration, you go to court in Santa Clara.

2. The TOS for GCP is very clear about an injunctive relief carve out:

> 14.11 Equitable Relief. Nothing in this Agreement will limit either party's ability to seek equitable relief.

3. This (according to the incident report) wasn't Railway being targeted specifically, it was a platform wide automated action. That's gross negligence, and the limitation of liability caps on Google's side of the agreement may not even be enforceable.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#379

Post by bastawhiz »

array_key_first wrote: post #366
bastawhiz wrote: post #358

Earlier quoted context omitted.

They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But…

I'm sure their contract explicitly states that their account can be suspended or terminated immediately without prior notice upon violating some TOS. And, most TOS are incredibly wide and vague, it wouldn't necessarily be hard to find something they violated. This is sort of the problem with these new-age internet companies. The contracts are incredibly hostile. Most TOS you see amount to "you have no rights and we c…

> it wouldn't necessarily be hard to find something they violated.

The incident report says this was a platform wide automated action. This argument falls apart.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#380

Post by sammy2255 »

zootboy wrote: post #281
sammy2255 wrote: post #152

The 3-2-1 backup rule is pretty outdated in the world of cloud. You could have 3 complete copies of your data in different S3 buckets, but if they're all under the same account you've lost your blast radius protection

It's not outdated, you just actually need to follow it. 3 copies of data in separate S3 buckets is ignoring the "2" in the 3-2-1 rule: 2 different mediums, and also the "1" rule: 1 copy offsite. In the cloud era, offsite means not on the same cloud provider. Different mediums ideally means a non-cloud provider (e.g. a NAS at your office under your control).

It certainly is outdated, by that very example you just provied. Why can't my two backups be on two separate S3 providers?
Post reply on HN