Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

91–100 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#91

Post by ndneighbor »

r_lee wrote: post #70
ndneighbor wrote: post #53

Earlier quoted context omitted.

> decisions are easy to criticize in hindsight I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project ge…

could you clarify, did an automated process by Google delete a GCP project/account/resource(s)? like, what exactly were you seeing when trying to get access or see what happened?

They deleted our GCP proj. sans warning. Still working the details, but that's how this whole thing began.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#92

Post by alchemism »

kevin_nisbet wrote: post #89
tjpnz wrote: post #59

Earlier quoted context omitted.

AWS normally contacts you first.

Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

I’m fairly certain you are supposed to contact any vendor before attempting to penetrate hosts with authorization, not the other way around.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#95

Post by jameson »

dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

What was the reason GCP took down a startup previously?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#96

Post by impulser_ »

brookst wrote: post #43
impulser_ wrote: post #23

Earlier quoted context omitted.

They literally own their own data centers. That's whats surprising about this. They are lying to their customers when they say they operate their own data center because obviously they don't if everyone's apps are down with GCP blocking their account.

Is it not possible that they own their own data center and have an unfortunate Google dependency? Obviously a fiasco but I’m not prepared to call them liars when it could be an honest mistake.

Then don't say your not a "Cloud on top of a cloud" provider.

They even made fun of cloud providers being down when AWS was down.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#97

Post by binarycleric »

How the heck do these things happen, especially with companies with huge monthly spend? At my last job we had some suspicious workloads running on AWS and our TAM reached out to us before taking any action. Who wants to bet this was some AI automation gone wrong and because GCP seems to be allergic to actually contacting a human to get a response, this just sits in some support queue that outsourced workers look at after a few hours just to give a canned response?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#99

Post by ttoinou »

r_lee wrote: post #68

seriously, is it possible to trust GCP with critical data/services at this point if you're not a billion dollar company? I'm exaggerating but someone said they got "auto banned" what if that happens to a small account which hosts some really important data/services there?

Railway isnt far from being a billion dollar company, no ?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#100

Post by xyzzy_plugh »

r_lee wrote: post #68

seriously, is it possible to trust GCP with critical data/services at this point if you're not a billion dollar company? I'm exaggerating but someone said they got "auto banned" what if that happens to a small account which hosts some really important data/services there?

I've managed several accounts with GCP over the years and I've always maintained a great relationship with our contacts there. Some of these accounts were quite small, on the order of Is Google's communication good? No, not particularly. The only way something like TFA happens is if the relationship is neglected (by one or both parties). I'm not saying Railway did something wrong, but there are usually many flags and opportunities to correct long before drastic actions.

I get the impression that Railway plays fast and loose with a lot of their limits and resources and that Google may not be a fan of that.

Edit: would also like to say that if you put all your resources in one GCP project you are going to have a bad time. If you organize stuff over many projects it is very unlikely that they will ever take account wide action. I've had issues with, for example, a particular tenant's behavior, but it never jeopardized the other tenants.

Post reply on HN