Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

111–120 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#111
post #95

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

What was the reason GCP took down a startup previously?

[deleted]

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#112

Does anyone know how this even happens inside the walls of google? Is it an automated process? How is such a (presumably) high revenue account just magically blocked without human intervention? I'm quite perplexed.

There would have been efforts to contact them, but it would have been via their contact method, aka the email they set it up with. Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation. Sign up for terms. It will get you payment terms!

Honestly still insane to nuke a high-volume client's business after a single payment issue. There would be no reason for Google to believe that a single hiccup like that is evidence that they won't get paid and have to cut account access immediately.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#114

How the heck do these things happen, especially with companies with huge monthly spend? At my last job we had some suspicious workloads running on AWS and our TAM reached out to us before taking any action. Who wants to bet this was some AI automation gone wrong and because GCP seems to be allergic to actually contacting a human to get a response, this just sits in some support queue that outsourced workers look at a…

Nothing surprises me with anything related to support on GCP. While we absolutely do not need them, I have been through no less than 12 different Account Executives over the last 6y and they're all ENTIRELY and COMPLETELY useless. They all introduce themselves, beg me to setup a meeting w/them and some sort of engineering resource(s), and they come to a meeting with a canned slide deck that is so absurdly unrelated t…

It's because they're measured on something, unsure which metric, but it's definitely not how helpful they are to you.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#115

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#116
post #59

Earlier quoted context omitted.

AWS normally contacts you first.

Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

Responding to an unknown security tester like that is a selling point, not a cautionary tale

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#117

How the heck do these things happen, especially with companies with huge monthly spend? At my last job we had some suspicious workloads running on AWS and our TAM reached out to us before taking any action. Who wants to bet this was some AI automation gone wrong and because GCP seems to be allergic to actually contacting a human to get a response, this just sits in some support queue that outsourced workers look at a…

Nothing surprises me with anything related to support on GCP. While we absolutely do not need them, I have been through no less than 12 different Account Executives over the last 6y and they're all ENTIRELY and COMPLETELY useless. They all introduce themselves, beg me to setup a meeting w/them and some sort of engineering resource(s), and they come to a meeting with a canned slide deck that is so absurdly unrelated t…

It doesn’t worry you enough that someday you could have a serious problem and they wouldn’t be able to help you?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#120

Earlier quoted context omitted.

> decisions are easy to criticize in hindsight I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project ge…

Thanks for getting things back up (genuinely mean that, btw). Upon logging back in I was prompted to promise I'm not deploying naughty things (I'm not). Was this in response to GCP detecting illegal (prohibited) behavior from something deployed via railway?

Actually, when I made the TOS check, I put that in Redis. That + the feature flags got reset.
Post reply on HN