Fly.io Postgres cluster down for 3 days, no word from them about it • Hacker News Board

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#411

Post by seti0Cha » Fri, Jul 21, 2023, 3:46 PM UTC

CoolCold wrote: post #403

seti0Cha wrote: post #396

Earlier quoted context omitted.

It seems to me like there's room for improving your customers' awareness around what is required for HA and how to tell when they are affected by a hardware issue. On the other hand, it may just be that the confusion is mostly amongst the casual onlookers, in which case you have my sympathies!

I'm not sure on this, will it make any sense - customers who DON'T WANT to be aware of what is required for HA (say lonely devs) choosing such a hosting types. Even if you put educational articles, I'm unsure it will be used. Putting some BANNER IN RED LETTERS into CLI output + link to article may work, though. What do you think?

I agree, articles tend not to get read by those who need them most. A warning from the CLI and a banner on the app management page with a link to a detailed explanation would seem like a good approach.

edit: sibling post shows there is such a message on the CLI. The only other thing I can think of is an "Are you sure you want to do this?" prompt, but in the end you can't reach everybody.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#412

Post by TheDong » Fri, Jul 21, 2023, 3:51 PM UTC

Jupe wrote: post #405

wgjordan wrote: post #397

Earlier quoted context omitted.

(Fly.io employee here) To clarify, we communicated this incident to the personalized status page [1] of all affected customers within 30 minutes of this single host going down, and resolved the incident on the status page once it was resolved ~47h later. Here's the timeline (UTC): - 2023-07-17 16:19 - host goes down - 2023-07-17 16:49 - issue posted to personalized status page - 2023-07-19 15:00 - host is fixed - 202…

Ouch? The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. For an entire cluster to be down for that long is just unacceptable. Rebuilding a cluster from the last-known-good backup should not take that long, unless there are PBs of data involved; dividing such large data stores into separate clusters/instances seems warranted. Solution archs should steer custom…

> The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days.

There was one physical server down. That's it. They even brought it back.

I've had AWS delete more instances, including all local NVMe store data, than I can count on my hands. Just in the last year.

Those instances didn't experience 47 hours downtime, they experienced infinite downtime, gone forever.

I guess by your standard I'd be fired for using AWS too.

But no, in reality, AWS deletes or migrates your instances all the time due to host hardware failure, and it's fine because if you know what you're doing, you have multiple instances across multiple AZs.

The same is true of fly. Sometimes underlying hardware fails (exactly like on AWS), and when that happens, you have to either have other copies of your app, or accept downtime.

I'll also add that the downtime is only 47 hours for you if you don't have the ability to spin up a new copy on a separate fly host or AZ in the meanwhile.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#413

Post by tptacek » Fri, Jul 21, 2023, 3:55 PM UTC

kunley wrote: post #406

tptacek wrote: post #379

Earlier quoted context omitted.

Apologies for repeating myself, but: You get to a certain number of servers and the probability on any one day that some server somewhere is going to hiccup and bounce gets pretty high. That's what happened here: a single host in Sydney, one of many, had a problem. When we have an incident with a single host, we update a notification channel for people with instances on that host. They are a tiny sliver of all our us…

> somehow a containerd boltdb on that host got corrupted, so when the machine bounced, containerd refused to come back online. We use containerd as a cache Hey, even if I can feel sympathetic for the course of unfortunate events, it's hard to not to comment: if you're using a cache, you should invalidate it on failure!

It's a read-through cache. This wasn't a cache invalidation issue. It's a systems-level state corruption problem that just happened to break a system used primarily as a cache.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#414

Post by kevin_nisbet » Fri, Jul 21, 2023, 4:03 PM UTC

tptacek wrote: post #390

CSSer wrote: post #385

Earlier quoted context omitted.

Why are your customers exposed to this? This sounds like a tough problem that I'm sympathetic to for you personally, but it sounds like there's no failover or appropriate redundancy in place to rollover to while you work to fix the problem. edit: I hope this comment doesn't sound accusatory. At the end of the day I want everyone to succeed. I hope there's a silver lining to this in the post-mortem.

The way to not be exposed to this is to run an HA configuration with more than one instance. If you're running an app on Fly.io without local durable storage, then it's easy to fail over to another server. But durable storage on Fly.io is attached NVMe storage. By far the most common way people use durable storage on Fly.io is with Postgres databases. If you're doing that on Fly.io, we automatically manage failover a…

Unless something has changed and I'm out of date, I think a piece of context here is fly postgres isn't really a managed service offering. From what I've seen fly does try to message this, but I think it's still easy for some subset of customers to miss that they're deploying an OSS component, maybe deployed a non-HA setup and forgot, and it's not the same as buying a database as a service.

So hopefully as fly.io get's more popular, there will be some compelling managed offerings. I saw comments at one point from the neon CEO about a fly.io offering, but not sure if that went anywhere. I'm sure customers can also use crunchy, or other offerings.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#415

Post by the_duke » Fri, Jul 21, 2023, 4:04 PM UTC

TheDong wrote: post #412

Jupe wrote: post #405

Earlier quoted context omitted.

Ouch? The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. For an entire cluster to be down for that long is just unacceptable. Rebuilding a cluster from the last-known-good backup should not take that long, unless there are PBs of data involved; dividing such large data stores into separate clusters/instances seems warranted. Solution archs should steer custom…

> The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. There was one physical server down. That's it. They even brought it back. I've had AWS delete more instances, including all local NVMe store data, than I can count on my hands. Just in the last year. Those instances didn't experience 47 hours downtime, they experienced infinite downtime, gone forever. I gue…

The core issue here is that fly doesn't offer distributed storage, only local disks.

Combine that with them having tooling for setting up Postgres built on top of single node storage, and you have the downtime problems and unhappy customers as a given.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#416

Post by yjftsjthsd-h » Fri, Jul 21, 2023, 4:06 PM UTC

TheDong wrote: post #412

Jupe wrote: post #405

Earlier quoted context omitted.

Ouch? The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. For an entire cluster to be down for that long is just unacceptable. Rebuilding a cluster from the last-known-good backup should not take that long, unless there are PBs of data involved; dividing such large data stores into separate clusters/instances seems warranted. Solution archs should steer custom…

> The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. There was one physical server down. That's it. They even brought it back. I've had AWS delete more instances, including all local NVMe store data, than I can count on my hands. Just in the last year. Those instances didn't experience 47 hours downtime, they experienced infinite downtime, gone forever. I gue…

When does AWS delete instances? Migrate, sure, and yes, local storage is supposed to be treated as disposable for that reason, but AFAIK only spot instances should be able to be destroyed outright.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#417

Post by CryptoBanker » Fri, Jul 21, 2023, 4:09 PM UTC

stephenr wrote: post #349

gtirloni wrote: post #302

Earlier quoted context omitted.

Was the sarcasm an attempt to discredit criticism of Fly's operational processes by pointing out that another company also has issues in how they handle outage notifications?

You could have just answered my previous question with "No, I am not familiar with sarcasm". Because you clearly don't understand sarcasm, I'll be blunt: No, I'm not trying to discredit any criticism of this provider. I agree with the comment I replied to, that this kind of failure mode is fucking ridiculous. My response thus is not an attempt to normalise this, but to highlight the elephant in the room, which is tha…

I don’t think you understand the comment you’re replying to at all. Either that or you’re trying to deflect even further

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#418

Post by mrkurt » Fri, Jul 21, 2023, 4:10 PM UTC

CSSer wrote: post #383

seti0Cha wrote: post #372

Earlier quoted context omitted.

Not a great summary from my perspective. Here's what I got out of it: - Their free tier support depended on noticing message board activity and they didn't. - Those experiencing outages were seeing the result of deploying in a non-HA configuration. Opinions differ as to whether they were properly aware that they were in that state. - They had an unusually long outage for one particular server. - Those points combined…

I'm surprised by your risk tolerance. If I had any cloud service at this level in my stack go down for three days, I'd start shopping for an alternative. This exceeds the level of acceptability for me for even non-HA requirements. After all, if I can't trust them for this, why would I ever consider giving them my HA business? Just based on napkin math for us, this could've been a potential loss of nearly half a milli…

You're saying a single server failure is going to to cost your business half a million dollars?

This was a server with local NVMe storage. The simplest thing to do would have been to just get rid of it, but we have quite a few free users with data they care about running on single node Postgres (because it's cheaper). It seemed like a better idea to recover this thing.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#419

Post by TheDong » Fri, Jul 21, 2023, 4:12 PM UTC

yjftsjthsd-h wrote: post #416

TheDong wrote: post #412

Earlier quoted context omitted.

> The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. There was one physical server down. That's it. They even brought it back. I've had AWS delete more instances, including all local NVMe store data, than I can count on my hands. Just in the last year. Those instances didn't experience 47 hours downtime, they experienced infinite downtime, gone forever. I gue…

When does AWS delete instances? Migrate, sure, and yes, local storage is supposed to be treated as disposable for that reason, but AFAIK only spot instances should be able to be destroyed outright.

To quote from their docs: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance...

> If your instance root device is an instance store volume, the instance is terminated, and cannot be used again.

See also the aws "Dedicated Hosts" and "Mac Instances". Those also have similar termination behavior.

The majority of my instances lost are from the instance store thing.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#420

Post by api » Fri, Jul 21, 2023, 4:16 PM UTC

Ensorceled wrote: post #357

TekMol wrote: post #198

Earlier quoted context omitted.

What exactly does Fly.io give you? You already know how to set up your project locally. Why not just do the same setup on any cloud VM and boom it is online?

Who fully sets up a significant project locally? I used Heroku for a project mostly because my team didn't have skill set to set this up and I wasn't going to do it. As far as I know they are still on Heroku (with a smattering of AWS services) for that same reason: just works and cheaper than doing it yourself.

> Who fully sets up a significant project locally?

Who doesn't? I couldn't imagine having to push to some cloud agent and wait a random amount of time every time I want to test something. With it local I can just save, maybe rebuild or have it auto-rebuild if necessary, and test, then repeat. On a fast machine this can be a few seconds or instantaneous.

Maybe the niche I'm missing here is very "green" developers who don't know how to do any sysadmin work or deploy things.

If this is you, learn it. It pays off huge, not just during development but in being able to have a lot more choice about where you deploy and a lot more control over your own stuff.