Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

131–140 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#131

There's a lot of, what seems to me, unfounded blame being directed at Google for this. Isn't railway the company that just blamed Anthropic for deleting their prod database?

Nope, Railway was the company who was hosting PocketOS, which is the company that blamed Cursor for deleting their prod database. Railway is only involved insofar as their API allowed an instant delete of the prod database.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#132

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

IIRC the Paris datacenter flood took down a whole “region” and some data was permanently unrecoverable.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#133

Is google allergic to humans or something? Cannot they just send an email or call the company before taking a wrecking ball to the entire company's infra? Are they stupid?

Surely this is automated. They wouldn't waste precious dollars on employing humans just to keep other humans happy.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#134

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

AWS goes down catastrophically but are back up in minutes/hours most of the time (as long as they aren't down because Iran blew up their data center). That's obviously REALLY bad for certain industries, but I suspect for the vast majority of their customers it's not a big deal. We've been able to isolate the damage almost every time just by having AZ failover in place and avoiding us-east-1 where we can.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#136

Well, as a 2 week tenured and very happy Railway customer until now, I am now a Render customer. Somehow DNS cut over within 1 min(!) and live after about 30 minutes of work. Not bad!

In my experience, DNS changes are a lot faster than they used to be. There’s some website that has a map that tries to resolve your domain with a bunch of name servers around the world that was pretty neat to look at last time I migrated something.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#137

There's a lot of, what seems to me, unfounded blame being directed at Google for this. Isn't railway the company that just blamed Anthropic for deleting their prod database?

fairly certain you are remembering the goofy article that was going around where a railway user allowed an agent to delete his db. iirc he questioned the agent after and the agent told him it should have read the file that told him not to do things, so just sounds like he deleted his db and blamed his tools.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#138

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

> Never heard of this from AWS or Azure.

AWS does it more efficiently; it takes down many startups at a time when us-east-1 goes down.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#139
post #128

Earlier quoted context omitted.

Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

You should not be conducting unauthorized penetration tests against third party infrastructure providers without permission. They have processes and systems and usually just wants a heads up of what you plan to test and t the duration / timestamps. Cuz otherwise you look like a threat actor. That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, tha…

>That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, that’s of course a totally different matter.

Sorry for being unclear, the vendor was attacking our organization only, and any other company was expressly forbidden in the contract. As I recall it was a fake SSO sign-in page to collect credentials that they would try and social engineer our employees with.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#140

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

> Never heard of this from AWS or Azure. AWS does it more efficiently; it takes down many startups at a time when us-east-1 goes down.

That’s an entirely different type of problem, and avoidable by just using us-east-2 (I still don’t understand why people default to us-east-1 unless they require some highly specific services).
Post reply on HN