Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

291–300 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#291

Post by swyx »

puppymaster wrote: post #274
tardwrangler wrote: post #248

Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.

another ditto from me, albeit anecdotal again. Railway dev teams play fast and loose with sprinkles of vibe coding everywhere on top. There's 'oops yea bear with us we are still a startup' and then there's railway.

i mean even google and aws are not without sin on this one. maybe wait for an RCA before punching someone who is currently down. theres a reason classy people do "hugops" when a competitor goes down, regardless of reputation.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#292

Post by swyx »

nikcub wrote: post #178
BitWiseVibe wrote: post #107

As someone who runs some public APIs, the amount of spam from Railway IPs is insane. They have horrible abuse prevention. Hopefully this encourages them to improve their operations.

This is the conflict at the center of running a hosting company - make it easy to signup and you get a lot of new users but also a lot of abuse. Implement anti-abuse measures and you will hit some loud false positives (this may be the case with GCP here). I don't envy anybody running a hosting co - the internet is a really ugly place under the surface. edit: to add - AWS are really good here. Must be the ~30 years of…

i wonder if DID or World (various ways of Proof of Human) can help solve this issue.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#293

Post by swyx »

twostorytower wrote: post #238
brokenodo wrote: post #119

Well, as a 2 week tenured and very happy Railway customer until now, I am now a Render customer. Somehow DNS cut over within 1 min(!) and live after about 30 minutes of work. Not bad!

I love pointing my name servers to Cloudflare so any DNS changes from that are practically instant.

as with many things, we say we like decentralization but quietly vote for centralization

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#294

Post by Kwpolska »

rajeshvar wrote: post #187
guluarte wrote: post #109

Earlier quoted context omitted.

It's Google. They let you use their services, but the moment you don't fit the norm, they suspend you.

What does blocked mean? Is there a different post that I am missing? There is shared infrastructure in GCP for networking (ex-googler here) and if only railway is affected, then it is not clear if it is only GCP or if there is something from Railway's perspective that needs to be addressed.

https://station.railway.com/community/what-we-know-so-far-ma...

> Around 22:20 UTC, our Google Cloud account was placed into a "restricted" status hence removing all of our cloud overflow VMs, our CloudSQL instance, and our API.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#295

Post by mlhpdx »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

None of the AWS “outages” have impacted us. They have either been regional, in which case we stand down the region (we run multiple hot regions), or didn’t involve things we need to maintain operation.

I can’t imagine AWS ever doing such a cascading delete. I mean, they have made deletion protection a difficult thing to ignore even for individual resources.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#296

Post by CodesInChaos »

oofbey wrote: post #158
jefborges wrote: post #78

Railway is back, but I’m not sure if I can trust keeping my projects there, so I’m going to migrate to another company.

After reading about how their delete database API also deletes all the backups, I concluded they are not to be trusted.

Don't all major clouds do that by default? But at least they have additional protections you can configure, if you know about them.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#298

Post by bootsmann »

nikcub wrote: post #178
BitWiseVibe wrote: post #107

As someone who runs some public APIs, the amount of spam from Railway IPs is insane. They have horrible abuse prevention. Hopefully this encourages them to improve their operations.

This is the conflict at the center of running a hosting company - make it easy to signup and you get a lot of new users but also a lot of abuse. Implement anti-abuse measures and you will hit some loud false positives (this may be the case with GCP here). I don't envy anybody running a hosting co - the internet is a really ugly place under the surface. edit: to add - AWS are really good here. Must be the ~30 years of…

Is it really a false positive if railway lets people run abusive services on GCP and then GCP consequently shuts them down?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#299

Post by rs_rs_rs_rs_rs »

tardwrangler wrote: post #248

Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.

>I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions

Sure but not even a warning before shutting down their account?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#300

Post by 1dom »

swyx wrote: post #291
puppymaster wrote: post #274

Earlier quoted context omitted.

another ditto from me, albeit anecdotal again. Railway dev teams play fast and loose with sprinkles of vibe coding everywhere on top. There's 'oops yea bear with us we are still a startup' and then there's railway.

i mean even google and aws are not without sin on this one. maybe wait for an RCA before punching someone who is currently down. theres a reason classy people do "hugops" when a competitor goes down, regardless of reputation.

Personally, I don't see this as people punching someone who's down. This is the sort of real life experience and necessary context from actual technical users that I come to HN comments for.

Someone is just asking to get Google's side and explaining why they want that, which seems reasonable since we're in a post where Google is being punched/blamed for this, and it sounds like it isn't Railways first questionable outage.

Post reply on HN