Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

141–150 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#141

Post by garciasn »

Rodeoclash wrote: post #126
YuriNiyazov wrote: post #122

Earlier quoted context omitted.

Don't know about GCP, but our AE on AWS was also continuously rotating, and as best I can tell, their job was to figure out what we are planning to build, and to ensure that we should always use for that, rather than a competitor product or build it ourselves.

Exactly the same experience for us as well. I just don't bother with them.

Before I just cut them off entirely, I used to tell them my primary concern was cost savings and that I wanted them to recommend ways I could cut 25% off my bill every month and watch the glorified salespeople fumble over trying to avoid that conversation.

It’s ok though, Claude helped us cut >45% of our monthly costs. I’m surprised they haven’t been beating down my door after we made that level-shift. Probably in AE transition. ¯\_(ツ)_/¯

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#142

Post by kevin_nisbet »

mixdup wrote: post #116
kevin_nisbet wrote: post #89

Earlier quoted context omitted.

Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

Responding to an unknown security tester like that is a selling point, not a cautionary tale

Yup, I thought it was great. Although one concern I always had in the back of my mind was where is the line drawn. Such as if an adversary gains access to one of my orgs accounts and does something similar, do we get 100% taken out.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#143

Post by jrockway »

r_lee wrote: post #68

seriously, is it possible to trust GCP with critical data/services at this point if you're not a billion dollar company? I'm exaggerating but someone said they got "auto banned" what if that happens to a small account which hosts some really important data/services there?

I don't think you can ever trust one service with critical data. Some Claude instance deletes your prod database, you have to restore from an offsite backup because it also deleted your local backups. Even at small startups we did pg_dump to AWS from GCP because ... who knows what is going to happen to GCP, and we want to continue to be in business if that happens.

I don't feel safe with any one single point of failure. "Your credit card bounced", "you thought it was dev", "you got hacked", etc. are all the same problem to me and no cloud provider solves those merely by setting up an account.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#145

Post by purduemike »

ndneighbor wrote: post #37
eoswald wrote: post #30

Earlier quoted context omitted.

Yep, and this is why I'm pissed. They lied. They're completely dependent on GCP. So, I gotta do some research, i need something a little more stable (and less dependent on one company's whims) than this. This is bad for them, because it really strikes at the heart of their 'big claim,' peacefull software deployments. This is chaos.

Yea, I mean, that's the whole MO of our platform and we failed at that. So yea, that's disappointing and more so for our customers. I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto…

Why CloudSQL? why not AlloyDB for stability?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#146

Post by ndneighbor »

binarycleric wrote: post #97

How the heck do these things happen, especially with companies with huge monthly spend? At my last job we had some suspicious workloads running on AWS and our TAM reached out to us before taking any action. Who wants to bet this was some AI automation gone wrong and because GCP seems to be allergic to actually contacting a human to get a response, this just sits in some support queue that outsourced workers look at a…

huh- I guess there are two HN submissions with meaningful replies...

I said this in the other thread, we got access to our account back, but even with a Account Rep. and a CSM on our account- it still took them a while to figure out what was going on.

I'm sure it could have been worse if we didn't have a rep on our account.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#147

Post by pixl97 »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

GCP never goes down because they banned all their customers.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#148

Post by hnburnsy »

From their founder on X...

"Absolutely. The Railway network is a mesh ring between AWS, GCP, and Metal

So: - High availability interconnects - High availability path routing between clouds - Database itself is high availability

However, Google's VPC itself is not. So we will add a shard to Metal and AWS"

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#149

Post by plandis »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

GCP has had outages. From a quick search it looks like they had a global outage less than a year ago:

https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#150

Post by hnburnsy »

hnburnsy wrote: post #148

From their founder on X... "Absolutely. The Railway network is a mesh ring between AWS, GCP, and Metal So: - High availability interconnects - High availability path routing between clouds - Database itself is high availability However, Google's VPC itself is not. So we will add a shard to Metal and AWS"

More here...

https://x.com/JustJake

Post reply on HN