Cloud platform dependencies are becoming a huge single point of failure
Incident Report: Railway Blocked by Google Cloud [resolved]
231–240 of 381 posts
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#232I thought Railway was building their own data centers? [0] > The fact of the matter is, you simply cannot build a cloud on someone else’s cloud. Indeed… [0] https://blog.railway.com/p/launch-week-02-welcome
Vercel seems to be pulling it off. So does PlanetScale, albeit for databases only. But everything’s a database.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#233Is google allergic to humans or something? Cannot they just send an email or call the company before taking a wrecking ball to the entire company's infra? Are they stupid?
Surely this is automated. They wouldn't waste precious dollars on employing humans just to keep other humans happy.
It surprises me there's not a manual review for $$$$ accounts. Speculation at this stage, but it's weird they would be put in the Recycle Bin like that.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#234Lesson learned: don't rely on a single hyperscaler, even (or especially) as a startup.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#235Earlier quoted context omitted.
I’m very sympathetic and understand that decisions are easy to criticize in hindsight but leaving your database in GCP while moving everything else to your own data centres seems so backwards I can’t even begin to imagine how that could happen. Was this really an intentional design decision?
I have exactly the same architecture. You can easily administer a postgres/mysql on your own infrastructure, but it's also the one thing where backups and availability are super strict. I can easily support multi-region in Google Cloud or AWS and that's way harder to do on-prem, and it's also hard to handle the replication story as safely as with Google Cloud. The hope is that GCP et al. give you safety and availabil…
I have kind of the same architecture. I host multiple dedicated servers and vps instances in the Hetzner "cloud", but all of these connect to a few hosted databases by Hetzners web hosting packages for like 20 bucks a month. It sounds insane, but the one thing that absolutely needs to stay online, is the database, so not hosting this myself makes sense. And since Hetzner is apparently tuned their dirt cheap databases pretty well, we can hammer them pretty hard without any problems.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#236Earlier quoted context omitted.
GCP has a lot of customers. But you wouldn't know the companies that do, unless you worked there and wanted to leak it, or it publicly comes out. Eg it's been publicly acknowledged that Apple uses GCP for iCloud, https://www.cnbc.com/amp/2018/02/26/apple-confirms-it-uses-g... , and Home Depot is another that's used as a case study, https://cloud.google.com/customers/the-home-depot but most customers don't want to mak…
upvoted & favourited because you taught me a really interesting fact which I feel makes up for an amazing discussion (regarding icloud using GCP). also, I can't help but imagine if instead of render, it was Apple's account which could've been auto-banned (Render is almost a billion dollar company or series-B, I am not sure) I haven't read the articles and I admit that but can you please elaborate to me on why Apple u…
> So in some sense if Apple is using gcp's for icloud then aren't they just reselling google storage themselves and google can always beat them in pricing while also wanting to chew away at the percentage of iphones themselves too?
Apple uses Samsung displays and Sony camera sensors, iirc, both of which are flagship Android phone makers. That doesn't really seem to be a concern in their procurement thinking. iCloud and Google Photos are not that direct competitors because which one is native depends on which phone you already bought. Google Photos definitely does have some market share on iOS due to having 3x the free storage and a handy compression mode (which used to be entirely unmetered at launch but now still uses storage, just less of it). But it will never be a full competitor because it is a separate app you have to install and it can't magically fetch cloud-only photos from the camera roll and photo picker UI like iCloud can.
The pricing of Google One and Apple One/iCloud+ isn't really dictated by underlying storage costs. At the higher tiers like 2TB, many don't come close to using all, while the laughable 5GB iCloud free tier clearly costs almost nothing in raw store, even on nVME SSD, if you compare it to S3/Backblaze or even raw disk pricing on the cloud.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#237Earlier quoted context omitted.
On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.
AWS goes down catastrophically but are back up in minutes/hours most of the time (as long as they aren't down because Iran blew up their data center). That's obviously REALLY bad for certain industries, but I suspect for the vast majority of their customers it's not a big deal. We've been able to isolate the damage almost every time just by having AZ failover in place and avoiding us-east-1 where we can.
> AWS goes down catastrophically but are back up in minutes/hours most of the time
The outage in the linked article appears to have been resolved in 4-5 hours.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#238Well, as a 2 week tenured and very happy Railway customer until now, I am now a Render customer. Somehow DNS cut over within 1 min(!) and live after about 30 minutes of work. Not bad!
I love pointing my name servers to Cloudflare so any DNS changes from that are practically instant.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#239Earlier quoted context omitted.
That’s an entirely different type of problem, and avoidable by just using us-east-2 (I still don’t understand why people default to us-east-1 unless they require some highly specific services).
Is it that easily avoidable? A lot of AWS's control plane seems to have dependencies on us-east-1, or at least that's what it's looked like as a non-us-east-1 user during recent outages.
I don't know how much it's improved, but a bunch of URLs they use unnecessarily have region specific details in them.
I remember a Workspaces outage about 5 or 6 years ago, and the problem for us was that the redirect link in the console had US East 1 in it.
The workspaces themselves weren't in US East 1 and nothing relied on US East 1.
Emailing users who needed it an alternative link with a different region in the URL for the login redirect fixed it for us.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#240Earlier quoted context omitted.
On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.
I still remember the one where they nuked all the storage of I think an Australian insurance company I think, luckily the it department had done a multi cloud setup for backups
Google Cloud accidentally deletes $125 billion Australian pension fund - May 2024