Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

121–130 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#121

Does anyone know how this even happens inside the walls of google? Is it an automated process? How is such a (presumably) high revenue account just magically blocked without human intervention? I'm quite perplexed.

There would have been efforts to contact them, but it would have been via their contact method, aka the email they set it up with. Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation. Sign up for terms. It will get you payment terms!

Yeah, I'm not sure what to think here. We know Google is not the best at customer service and has automated account suspensions. But, what I'm curious about here is why this happened.

Railway hosts applications for customers. An uneducated guess for some possible reasons: 1) one of those customers hosted something they shouldn't have 2) railway had something spawn that took up too many resources 3) Or their account balance was too high 4) Or something...

But all of this probably culminates in someone needed to read an email that was missed.

Scaling a customer infrastructure setup like Railway is hard. This is one of the non-technical hard parts - how to make sure your account with your primary vendor is safe. But, I'm willing to wait to pass judgement here until more information is available. I'm sure the post-mortem will have lessons. I'd like to know more.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#122

Earlier quoted context omitted.

Nothing surprises me with anything related to support on GCP. While we absolutely do not need them, I have been through no less than 12 different Account Executives over the last 6y and they're all ENTIRELY and COMPLETELY useless. They all introduce themselves, beg me to setup a meeting w/them and some sort of engineering resource(s), and they come to a meeting with a canned slide deck that is so absurdly unrelated t…

It's because they're measured on something, unsure which metric, but it's definitely not how helpful they are to you.

Don't know about GCP, but our AE on AWS was also continuously rotating, and as best I can tell, their job was to figure out what we are planning to build, and to ensure that we should always use for that, rather than a competitor product or build it ourselves.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#123

Earlier quoted context omitted.

Nothing surprises me with anything related to support on GCP. While we absolutely do not need them, I have been through no less than 12 different Account Executives over the last 6y and they're all ENTIRELY and COMPLETELY useless. They all introduce themselves, beg me to setup a meeting w/them and some sort of engineering resource(s), and they come to a meeting with a canned slide deck that is so absurdly unrelated t…

It doesn’t worry you enough that someday you could have a serious problem and they wouldn’t be able to help you?

On the list of things that worry me the most about our company's stuff, an issue I cannot solve w/o help from a human at GCP is around #900000042.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#125

one of the many reasons companies are cloud agnostic and dont want to get locked in

Yeah but until you find that the new cloud provider won't approve your compute quota or doesn't have enough capacity in the region or you hit fraud flags for stagnant account spinning up lots of compute.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#126

Earlier quoted context omitted.

It's because they're measured on something, unsure which metric, but it's definitely not how helpful they are to you.

Don't know about GCP, but our AE on AWS was also continuously rotating, and as best I can tell, their job was to figure out what we are planning to build, and to ensure that we should always use for that, rather than a competitor product or build it ourselves.

Exactly the same experience for us as well. I just don't bother with them.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#127

Earlier quoted context omitted.

Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

I’m fairly certain you are supposed to contact any vendor before attempting to penetrate hosts with authorization, not the other way around.

Having done this for both Azure and AWS, there's a specific ticket that needs to be filed with each provider that documents the scope of your pen test, where you're coming from, and a time frame over which you're doing it (which ISTR was "not more than 24 hours")

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#128
post #59

Earlier quoted context omitted.

AWS normally contacts you first.

Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

You should not be conducting unauthorized penetration tests against third party infrastructure providers without permission. They have processes and systems and usually just wants a heads up of what you plan to test and t the duration / timestamps.

Cuz otherwise you look like a threat actor.

That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, that’s of course a totally different matter.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#129
post #95

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

What was the reason GCP took down a startup previously?

hn.algolia.com gcp blocked

https://news.ycombinator.com/item?id=46731498 https://news.ycombinator.com/item?id=33360416

Then I recall https://news.ycombinator.com/item?id=45798827

https://news.ycombinator.com/item?id=33737577

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#130

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

Yep, we also don't touch them for this same reason.
Post reply on HN