Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

191–200 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#191

Post by wmf »

Genego wrote: post #183
oofbey wrote: post #161

Earlier quoted context omitted.

Railway deserves a lot of blame here. Deleting backups along with the database is a lot like not having backups. Moronic design choice.

Why does Railway deserve any blame here at all? It was an MCP with elevated infra access, that the user willingly connected through Cursor, which allowed an LLM Agent to manage infra on Railway. The user would first have gone through oAuth confirming the access level scope (I would have rejected the moment it indicates to me that it can delete critical infra and backups...). So obviously it has access to all commands…

Putting AI aside, people make mistakes. One of the most common mistakes people make is deleting the wrong thing. After they realize the mistake, people want to restore the thing they deleted from backups. Thus deleting the thing and deleting the backups of the thing should always be separate operations.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#192

Post by erikerikson »

andreareina wrote: post #184
abofh wrote: post #155

Earlier quoted context omitted.

I've been in AWS for almost twenty years at this point. It's been a long time since I've seen a global outage of the data plane on anything. The control plane, especially the US-east-1 services? Yes - but if you're off of east-1, your outages are measured in missile strikes, not botched deployments.

Didn't the latest outage affect people not on us-east-1 because internal aws services depend on us-east-1?

The impacts are usually partial. For example, scaling is impacted but everything already deployed contributes to work up to capacity. Or, you can't change configuration but the previous configuration works as configured. Often surprisingly not so impactful even if there can be limited work stoppage.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#194

Post by yandie »

overfeed wrote: post #138
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

> Never heard of this from AWS or Azure. AWS does it more efficiently; it takes down many startups at a time when us-east-1 goes down.

During my 5 years of my startup, we had only 1 outage due to AWS because we picked us-west-2 as the primary reason. If anyone starting a company and picks us-east-1 as the primary reason, they should be fired. There's absolutely no reason to be in that region.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#195

Post by nbarbettini »

DrewADesign wrote: post #136
brokenodo wrote: post #119

Well, as a 2 week tenured and very happy Railway customer until now, I am now a Render customer. Somehow DNS cut over within 1 min(!) and live after about 30 minutes of work. Not bad!

In my experience, DNS changes are a lot faster than they used to be. There’s some website that has a map that tries to resolve your domain with a bunch of name servers around the world that was pretty neat to look at last time I migrated something.

I became so conditioned to waiting hours(!) for DNS propagation that I'm always pleasantly surprised when it takes <5 min these days.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#197

Post by kvakvs »

valgaze wrote: post #182

May 2024 UniSuper incident: https://cloud.google.com/blog/products/infrastructure/detail... https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... A joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian 8 May 2024 UniSuper and Google Cloud understand the disruption to services experienced by members has been extremely frustrating and disappointing. We extend our sincere apologies to…

The instant cascading worldwide deletion upon closing or deleting a subscription sounds like a recipe for disaster. Why not mark it for deletion and delete say... a day or a week later?

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#198

Post by tempest_ »

yandie wrote: post #194
overfeed wrote: post #138

Earlier quoted context omitted.

> Never heard of this from AWS or Azure. AWS does it more efficiently; it takes down many startups at a time when us-east-1 goes down.

During my 5 years of my startup, we had only 1 outage due to AWS because we picked us-west-2 as the primary reason. If anyone starting a company and picks us-east-1 as the primary reason, they should be fired. There's absolutely no reason to be in that region.

Why do people want to be in that region? Is it the default or something?

I know some workloads help to be colocated but all these places are connected by fiber and every cloud has a worldwide CDN it seems.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#199

Post by xavdid »

overfeed wrote: post #138
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

> Never heard of this from AWS or Azure. AWS does it more efficiently; it takes down many startups at a time when us-east-1 goes down.

If my cloud provider brings my startup down, it's my problem. If they bring all the startups down, that's their problem.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#200

Post by Shank »

kevin_nisbet wrote: post #139
dannyw wrote: post #128

Earlier quoted context omitted.

You should not be conducting unauthorized penetration tests against third party infrastructure providers without permission. They have processes and systems and usually just wants a heads up of what you plan to test and t the duration / timestamps. Cuz otherwise you look like a threat actor. That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, tha…

>That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, that’s of course a totally different matter. Sorry for being unclear, the vendor was attacking our organization only, and any other company was expressly forbidden in the contract. As I recall it was a fake SSO sign-in page to collect credentials that they would try and social engineer our empl…

At a minimum you should contact AWS before you launch a phishing page as a test that targets AWS customers.
Post reply on HN