Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

301–310 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#301

Post by GoblinSlayer »

suttontom wrote: post #269
dantiberian wrote: post #221

Earlier quoted context omitted.

I wrote about the UniSuper issue at the time: https://danielcompton.net/google-cloud-unisuper . It was a pretty nasty bug where their VMWare environment was created with a one-year expiry date, but was one "resource" from the perspective of Google Cloud.

"UniSuper’s production Google Cloud VMware Engine (GCVE) private cloud was automatically deleted one year after it’s creation due to a misconfiguration in how it was created. When it was created, there was a bug in the creation script which passed a null value." That's pretty amazing. Not due to a cascading failure from someone changing a config deep inside of a system that caused a bunch of unintended effects, just…

Probably javascript. Shell scripts don't have null values.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#302

Post by duckmysick »

nikcub wrote: post #178
BitWiseVibe wrote: post #107

As someone who runs some public APIs, the amount of spam from Railway IPs is insane. They have horrible abuse prevention. Hopefully this encourages them to improve their operations.

This is the conflict at the center of running a hosting company - make it easy to signup and you get a lot of new users but also a lot of abuse. Implement anti-abuse measures and you will hit some loud false positives (this may be the case with GCP here). I don't envy anybody running a hosting co - the internet is a really ugly place under the surface. edit: to add - AWS are really good here. Must be the ~30 years of…

Hetzner is famously aggressive with their KYC (Know Your Customer) requirements, often locking new sign-ups and asking for photos of ID.

Damned if you do, damned if you don't.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#303

Post by nullsanity »

1dom wrote: post #300
swyx wrote: post #291

Earlier quoted context omitted.

i mean even google and aws are not without sin on this one. maybe wait for an RCA before punching someone who is currently down. theres a reason classy people do "hugops" when a competitor goes down, regardless of reputation.

Personally, I don't see this as people punching someone who's down. This is the sort of real life experience and necessary context from actual technical users that I come to HN comments for. Someone is just asking to get Google's side and explaining why they want that, which seems reasonable since we're in a post where Google is being punched/blamed for this, and it sounds like it isn't Railways first questionable ou…

[dead]

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#304

Post by GoblinSlayer »

jedberg wrote: post #267
Alive-in-2025 wrote: post #222

Earlier quoted context omitted.

I wonder if big companies can get a special contract with something like you can't delete my service automatically (unless it's an emergency)

If you're big enough you don't need a contract for that, that's just their default method of operation.

#48203226

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#305

Post by tauntz »

astafrig wrote: post #159
usernametaken29 wrote: post #144

I didn’t knew Railway so with this misleading headline I thought a Google Cloud data centre was being built in the way of a railroad. That’d been a funny story to read..

How is the title misleading?

"Railway Blocked by Google Cloud"

If you don't happen to know that "Railway" is referring to a company, then you might reasonably read that as "a GCP outage caused issues in the train network somewhere".

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#306

Post by HighGoldstein »

erikerikson wrote: post #192
andreareina wrote: post #184

Earlier quoted context omitted.

Didn't the latest outage affect people not on us-east-1 because internal aws services depend on us-east-1?

The impacts are usually partial. For example, scaling is impacted but everything already deployed contributes to work up to capacity. Or, you can't change configuration but the previous configuration works as configured. Often surprisingly not so impactful even if there can be limited work stoppage.

Considering how many AWS and non-AWS services go down at least partially when us-east-1 fails, this reads somewhat like "Don't worry that the steering wheel and pedals aren't working, your engine is still running on cruise control".

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#308

Post by shamcleren »

blobbers wrote: post #162
somewhatgoated wrote: post #115

Earlier quoted context omitted.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

Unfortunately, if everyone goes down people are understanding. If just _you_ go down, then its oddly less forgiveable.

[dead]

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#309

Post by egorfine »

rs_rs_rs_rs_rs wrote: post #299
tardwrangler wrote: post #248

Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.

>I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions Sure but not even a warning before shutting down their account?

First time?

It's google, come on.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#310

Post by chillfox »

shye wrote: post #203
kvakvs wrote: post #197

Earlier quoted context omitted.

The instant cascading worldwide deletion upon closing or deleting a subscription sounds like a recipe for disaster. Why not mark it for deletion and delete say... a day or a week later?

From personal experience, as a customer who once did something stupid: Google Cloud does soft deletes. But you need to reach out to support fast enough. And really, if you deleted something important and discovered it only the next day, and not within minutes, you're having a bigger issue that a soft delete won't solve.

What kind of shitty soft delete can’t be undone a few weeks after?

Weekends and public holidays are a thing, plus it’s quite common for companies to shut down for 2 weeks over Christmas.

There’s a lot of opportunity for mistakes or malicious actions to happen at times that won’t be discovered for a while.

Post reply on HN