Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report: Railway Blocked by Google Cloud [resolved]

status.railway.com

Post reply on HN
Print view View original

151–160 of 381 posts

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#151

Post by devmor »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

There was a pretty bad one last summer - their IAM system got a bad update and it broke almost all GCP services for an hour or so, since every authenticated API call reaches out to IAM.

It had lasting effects for us for a little over 3 hours.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#153

Post by [deleted] »

kevin_nisbet wrote: post #89
tjpnz wrote: post #59

Earlier quoted context omitted.

AWS normally contacts you first.

Do they? The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

[deleted]

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#154

Post by faangguyindia »

MrDarcy wrote: post #77
whh wrote: post #51

This could kill a startup. I really don't like Google's automated and silent account murder functionality.

There’s no way this was automated or silent. The only reasonable explanation is Railway lost control of their estate and something was happening that warranted a group of humans to decide flipping the kill switch was the best of a set of bad alternatives.

you can go on google cloud subreddit and watch horror stories

i actually built a good plan out of those horror stories for my companies.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#155

Post by abofh »

somewhatgoated wrote: post #115
dangoodmanUT wrote: post #39

It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.

On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.

I've been in AWS for almost twenty years at this point. It's been a long time since I've seen a global outage of the data plane on anything. The control plane, especially the US-east-1 services? Yes - but if you're off of east-1, your outages are measured in missile strikes, not botched deployments.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#158

Post by oofbey »

jefborges wrote: post #78

Railway is back, but I’m not sure if I can trust keeping my projects there, so I’m going to migrate to another company.

After reading about how their delete database API also deletes all the backups, I concluded they are not to be trusted.

Re: Incident Report: Railway Blocked by Google Cloud [resolved]

#160

Post by rsync »

sammy2255 wrote: post #152

The 3-2-1 backup rule is pretty outdated in the world of cloud. You could have 3 complete copies of your data in different S3 buckets, but if they're all under the same account you've lost your blast radius protection

If only there were a quick and easy way to replicate s3 buckets to an independent provider…

… on the Unix command line …

… to a cloud older than AWS…

… if only …

Post reply on HN