Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

281–290 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#281

Post by zootboy »

sammy2255 wrote: post #152

The 3-2-1 backup rule is pretty outdated in the world of cloud. You could have 3 complete copies of your data in different S3 buckets, but if they're all under the same account you've lost your blast radius protection

It's not outdated, you just actually need to follow it. 3 copies of data in separate S3 buckets is ignoring the "2" in the 3-2-1 rule: 2 different mediums, and also the "1" rule: 1 copy offsite. In the cloud era, offsite means not on the same cloud provider. Different mediums ideally means a non-cloud provider (e.g. a NAS at your office under your control).

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#282

Post by onion2k »

raverbashing wrote: post #270
dantiberian wrote: post #221

Earlier quoted context omitted.

I wrote about the UniSuper issue at the time: https://danielcompton.net/google-cloud-unisuper . It was a pretty nasty bug where their VMWare environment was created with a one-year expiry date, but was one "resource" from the perspective of Google Cloud.

Creating stuff with 1yr (implicit) expiry by default is just a delayed footgun tbh

That's one footgun, but then pushing that into production and actually deleting things rather than queuing them to be deleted later after a sanity check until the system is stable, and not informing users that the 1 year policy existing, (probably) not documenting that the expiry exists, not testing 'what happens if we pass in null?', etc are a whole series of mistakes.

This was less "Oh look, a rare edge case that was easy to miss!" and more "We don't bother putting guardrails into critical systems. Oops!"

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#283

Post by IshKebab »

suttontom wrote: post #269
dantiberian wrote: post #221

Earlier quoted context omitted.

I wrote about the UniSuper issue at the time: https://danielcompton.net/google-cloud-unisuper . It was a pretty nasty bug where their VMWare environment was created with a one-year expiry date, but was one "resource" from the perspective of Google Cloud.

"UniSuper’s production Google Cloud VMware Engine (GCVE) private cloud was automatically deleted one year after it’s creation due to a misconfiguration in how it was created. When it was created, there was a bug in the creation script which passed a null value." That's pretty amazing. Not due to a cascading failure from someone changing a config deep inside of a system that caused a bunch of unintended effects, just…

This is why you never use shell scripts for non-interactive tasks.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#284

Post by danpalmer »

shooker435 wrote: post #57
danpalmer wrote: post #21

Earlier quoted context omitted.

> It's strange such a big corp can't even afford to have proper support team Railway say they are in touch with that support team.

god help them

I had good experiences with their support, and bad experiences with AWS support. tldr: YMMV.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#285

Post by trvz »

Avicebron wrote: post #29

Isn't Railway the "the API key to delete the backups is in the prod database, because that's where the backups live duh" guys?

No, this is the company that failed those guys.

You should also read the story, as you're perpetuating a false version of it: https://x.com/lifeof_jer/status/2048103471019434248

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#286

Post by onion2k »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

You can't have 100% uptime. It's unfeasible, especially for a startup. You should be telling your customers that downtime might happen, sometimes for reasons beyond your control, and that if it does then you'll do your best to recover and to compensate them for the inconvenience. You should cultivate a relationship with your early customers that makes them feel bad for you when there's an outage rather than angry about how it impacts them. Maybe even go as far as firing the customers who give you a hard time over it. That way if your cloud provider falls over it's really annoying but not a big deal.

Your cloud provider blocking your business from running is far worse.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#287

Post by raverbashing »

locknitpicker wrote: post #273
raverbashing wrote: post #256

Earlier quoted context omitted.

No, that's the naive view Because in case of a compromise/unauthorized access that's exactly what you don't want to happen

> No, that's the naive view No, not really. That's pretty basic stuff. You would do well in reading up on the shared responsibility model. Customers are responsible for setting up their own infrastructure, and platform/service providers are only responsible for the services they manage. Even then, stuff like persisted data is still recoverable by design. But you are absolutely responsible for the service you put toge…

Oh "reading about it"?

Try experiencing it in person

Again this is the naive view

Again if someone compromises your accounts and everything is deleted instantly you'll be the one looking like a fool

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#290

Post by realityking »

garciasn wrote: post #141
Rodeoclash wrote: post #126

Earlier quoted context omitted.

Exactly the same experience for us as well. I just don't bother with them.

Before I just cut them off entirely, I used to tell them my primary concern was cost savings and that I wanted them to recommend ways I could cut 25% off my bill every month and watch the glorified salespeople fumble over trying to avoid that conversation. It’s ok though, Claude helped us cut >45% of our monthly costs. I’m surprised they haven’t been beating down my door after we made that level-shift. Probably in AE…

My experience with a large-ish ($5m/year) AWS account was quite different. They were happy to support us with cost optimizations, discounts, and one time credits for certain activities (co-innovation and archiving certain milestones in their partner program).

Their primary concern seemed to have been to keep as much of our workload inside AWS as possible and to win workload from 3rd party services we used (e.g. CDNs). The actual revenue appeared secondary.

Post reply on HN