Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

241–250 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#241

I didn’t knew Railway so with this misleading headline I thought a Google Cloud data centre was being built in the way of a railroad. That’d been a funny story to read..

An elevated railroad once ran through one end of what is now a Google-owned building (Chelsea Market in Manhattan). It's now part of the High Line elevated pedestrian park.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#242
post #178

As someone who runs some public APIs, the amount of spam from Railway IPs is insane. They have horrible abuse prevention. Hopefully this encourages them to improve their operations.

This is the conflict at the center of running a hosting company - make it easy to signup and you get a lot of new users but also a lot of abuse. Implement anti-abuse measures and you will hit some loud false positives (this may be the case with GCP here). I don't envy anybody running a hosting co - the internet is a really ugly place under the surface. edit: to add - AWS are really good here. Must be the ~30 years of…

I continue to receive phishing via AWS pretending to be Amazon. And not even the Unicode-lookalike shenanigans that my spam filter refuses for excessive mixed scripts, no; literally claiming to be Amazon as in: the company that operates the relay.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#243

Lesson learned: don't rely on a single hyperscaler, even (or especially) as a startup.

I just... I don't really understand why startups even use AWS, GC, or any other cloud hosted software? Hetzner, etc. Are all extremely cheap, and honestly scale so well... Code nowadays is cheaper for configs, and having full control over your compute is... liberating.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#244

Lesson learned: don't rely on a single hyperscaler, even (or especially) as a startup.

I just... I don't really understand why startups even use AWS, GC, or any other cloud hosted software? Hetzner, etc. Are all extremely cheap, and honestly scale so well... Code nowadays is cheaper for configs, and having full control over your compute is... liberating.

Oh absolutely... and many use architectures that have evolved out of the needs of really big companies and are not really a good fit for a startup. But I guess they want to be "ready for growth".

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#245

Earlier quoted context omitted.

GCP has a lot of customers. But you wouldn't know the companies that do, unless you worked there and wanted to leak it, or it publicly comes out. Eg it's been publicly acknowledged that Apple uses GCP for iCloud, https://www.cnbc.com/amp/2018/02/26/apple-confirms-it-uses-g... , and Home Depot is another that's used as a case study, https://cloud.google.com/customers/the-home-depot but most customers don't want to mak…

upvoted & favourited because you taught me a really interesting fact which I feel makes up for an amazing discussion (regarding icloud using GCP). also, I can't help but imagine if instead of render, it was Apple's account which could've been auto-banned (Render is almost a billion dollar company or series-B, I am not sure) I haven't read the articles and I admit that but can you please elaborate to me on why Apple u…

Firstly, apple doesn’t compete on price. Even if icloud is priced more than google people would always buy apple just for the ecosystem integration. It’s not even a competition to be honest.

Look up “buy or build” which is the industry term for this kind of evaluation: buy product and use it/resell it or build your own.

Apple has gone for different strategies in various areas:

Build own Apple silicon chips, do not buy off the shelf chips from intel or nvidia or amd.

Buy and resell google storage but don’t want to build their own distributed data store for end users.

It’s about what matters more for the company and the core products. Apple’s laptops, cell phones are considered core products. Icloud is a value add.

This is also why apple is making their own cell phone broadband chips. For most companies, this is a “buy from qualcolm” but apple needs to build their own for independence for their number 1 core product: the iphone.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#247

Lesson learned: don't rely on a single hyperscaler, even (or especially) as a startup.

I just... I don't really understand why startups even use AWS, GC, or any other cloud hosted software? Hetzner, etc. Are all extremely cheap, and honestly scale so well... Code nowadays is cheaper for configs, and having full control over your compute is... liberating.

Low cost to entry, easy to get scale from the beginning if you need it. The large cloud providers throw free credit at startups to lock them in all the time. I had a short lived stint trying to get my own startup off the ground and it was really easy to get free compute from Google with no strings attached. This was many years ago now, but I would be surprised if it is any different.

I am with you entirely and would not have taken that route today, but it is really easy to see why people go that route.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#248
Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence.

Regardless of how it happened, for me, this is the straw that broke the camel's back.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#249

Earlier quoted context omitted.

There would have been efforts to contact them, but it would have been via their contact method, aka the email they set it up with. Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation. Sign up for terms. It will get you payment terms!

Honestly still insane to nuke a high-volume client's business after a single payment issue. There would be no reason for Google to believe that a single hiccup like that is evidence that they won't get paid and have to cut account access immediately.

Railway might not be even in the realm of high-volume clients for Google. For all we know they might be efficient in utilizing Google infrastructure.

But most likely, it's just automations in place without an appropriate human override coupled with gross negligence.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#250

Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.

Could you point us to any specific past instances? I'd be interested to read about them.
Post reply on HN