Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

161–170 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#161

Post by oofbey »

mmmore wrote: post #131
parineum wrote: post #118

There's a lot of, what seems to me, unfounded blame being directed at Google for this. Isn't railway the company that just blamed Anthropic for deleting their prod database?

Nope, Railway was the company who was hosting PocketOS, which is the company that blamed Cursor for deleting their prod database. Railway is only involved insofar as their API allowed an instant delete of the prod database.

Railway deserves a lot of blame here. Deleting backups along with the database is a lot like not having backups. Moronic design choice.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#162

Post by blobbers »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

Unfortunately, if everyone goes down people are understanding. If just _you_ go down, then its oddly less forgiveable.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#163

Post by dylanpyle »

garciasn wrote: post #106
binarycleric wrote: post #97

How the heck do these things happen, especially with companies with huge monthly spend? At my last job we had some suspicious workloads running on AWS and our TAM reached out to us before taking any action. Who wants to bet this was some AI automation gone wrong and because GCP seems to be allergic to actually contacting a human to get a response, this just sits in some support queue that outsourced workers look at a…

Nothing surprises me with anything related to support on GCP. While we absolutely do not need them, I have been through no less than 12 different Account Executives over the last 6y and they're all ENTIRELY and COMPLETELY useless. They all introduce themselves, beg me to setup a meeting w/them and some sort of engineering resource(s), and they come to a meeting with a canned slide deck that is so absurdly unrelated t…

For what it's worth - I'm not sure what the criteria is (I assume we're "medium sized / not a big upsell opportunity"?) - our GCP rep quickly pushed us to switching to using a GCP reseller. They took over our billing so that we can pay via ACH, and provide both free first-line support/escalation and paid engagements for bigger projects; they don't charge a premium on top, apparently Google pays them for supporting us. Hasn't made much of a difference in how we operate, but at least we have a direct-ish line for issues when they come up.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#165

Post by Alive-in-2025 »

throwaranay4933 wrote: post #3

This screenshot from Discord suggests the idea that the outage is caused by automated GCP account ban: https://x.com/acgfbr/status/2056866780866351323

Automated account bans are the bane of internet existence today. I was banned from reddit for "bad behavior", I appealed and both times it's oops, there was nothing there, some automated system thought your comment was rude even though it wasn't.

Then they send you very strongly worded messages that says trying to work around the ban will lead to something bad happening.

I've been worried my main email account provider would do this. The core issue is even if you pay, even if you are a company as shown here companies don't carefully enough have limits on banning. I can only imagine they ban lots of scammy things every day so "they think it's working great".

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#166

Post by busterarm »

dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

Hetzner and OVH also do this all the time.

It's AWS and Azure that are the outliers and tend not to care too much what their customers do with their infrastructure. AWS is perfectly fine with allowing me to run copies of 15 year old vulnerable AMIs copied from AMIs they've long since deprecated and removed. Even for removed features like NAT AMIs.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#167

Post by danesparza »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

You can read the parent post, right?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#168

Post by MattGaiser »

stingraycharles wrote: post #140
overfeed wrote: post #138

Earlier quoted context omitted.

> Never heard of this from AWS or Azure. AWS does it more efficiently; it takes down many startups at a time when us-east-1 goes down.

That’s an entirely different type of problem, and avoidable by just using us-east-2 (I still don’t understand why people default to us-east-1 unless they require some highly specific services).

Sympathy. Railway is going to have numerous people blaming them for this outage. When us-east-1 fails, it is headline news, so you are not to blame.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#169

Post by aloha2436 »

stingraycharles wrote: post #140
overfeed wrote: post #138

Earlier quoted context omitted.

> Never heard of this from AWS or Azure. AWS does it more efficiently; it takes down many startups at a time when us-east-1 goes down.

That’s an entirely different type of problem, and avoidable by just using us-east-2 (I still don’t understand why people default to us-east-1 unless they require some highly specific services).

Is it that easily avoidable? A lot of AWS's control plane seems to have dependencies on us-east-1, or at least that's what it's looked like as a non-us-east-1 user during recent outages.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#170

Post by Izikiel43 »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

I still remember the one where they nuked all the storage of I think an Australian insurance company I think, luckily the it department had done a multi cloud setup for backups
Post reply on HN