Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

401–410 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#401

Post by creeble »

bjord wrote: post #355
danielvaughn wrote: post #56

Earlier quoted context omitted.

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

historically, I've used Vultr, but I don't see anyone talking about it—I'm curious if anyone else has thoughts on them? (I've been happy, but then again my usage has been exceedingly basic)

Have used both DO and Vultr for years. Put simply, DO is better, but Vultr isn’t terrible.

Higher number of outages at Vultr over 5 years, but none longer than a few hours. I can’t remember the last DO outage lasting more than a few minutes.

Experienced a Vultr routing problem that lasted several hours; they communicated about it, but it was still a long time to fix.

DO once did an auto-migration of a server to another cluster with an attendant outage that lasted a few minutes at most. No IP changes, completely transparent.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#402

Post by urduntupu »

tptacek wrote: post #392
Aurornis wrote: post #386

Earlier quoted context omitted.

> If you're reading my comments on HN as some kind of official response from the company, you've misconstrued them. For what it’s worth, this is the reason most companies eventually restrict their employees from making statements about the company; It doesn’t matter if you thought it was clear that is was unofficial, any statement from an employee in a position of power (such as someone with access to the control pan…

Maybe I'll get restricted someday!

For the sake of fly.io, you should either restrict yourself and not respond or, if you can't resist, make it crystal clear, that you DO NOT represent fly.io. Your first message can and will be misunderstood and it DOES throw a poor light on fly.io.

I am a paying customer of fly.io, on the Scale plan.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#403

Post by CoolCold »

seti0Cha wrote: post #396
tptacek wrote: post #379

Earlier quoted context omitted.

Apologies for repeating myself, but: You get to a certain number of servers and the probability on any one day that some server somewhere is going to hiccup and bounce gets pretty high. That's what happened here: a single host in Sydney, one of many, had a problem. When we have an incident with a single host, we update a notification channel for people with instances on that host. They are a tiny sliver of all our us…

It seems to me like there's room for improving your customers' awareness around what is required for HA and how to tell when they are affected by a hardware issue. On the other hand, it may just be that the confusion is mostly amongst the casual onlookers, in which case you have my sympathies!

I'm not sure on this, will it make any sense - customers who DON'T WANT to be aware of what is required for HA (say lonely devs) choosing such a hosting types. Even if you put educational articles, I'm unsure it will be used. Putting some BANNER IN RED LETTERS into CLI output + link to article may work, though.

What do you think?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#404

Post by TheDong »

CSSer wrote: post #400
tptacek wrote: post #390

Earlier quoted context omitted.

The way to not be exposed to this is to run an HA configuration with more than one instance. If you're running an app on Fly.io without local durable storage, then it's easy to fail over to another server. But durable storage on Fly.io is attached NVMe storage. By far the most common way people use durable storage on Fly.io is with Postgres databases. If you're doing that on Fly.io, we automatically manage failover a…

I see. Have you considered eliminating this configuration from your offering? It sounds like the terminology could confuse people, and it may be the case that they're assuming that a host isn't really what it is (a single host). This kind of thing is difficult for those seeking to build managed services, because I think people expect you to provide offerings that can't harm them when the cause is related to the servi…

Plenty of people would rather take downtime than pay for redundancy, for example for a test database.

AWS RDS lets you spin up a RDS instance that costs 3x less and regularly has downtime (the 'single-az' one), quite similar to this.

Anyone who's used servers before knows "A single instance" is the same as "sometimes you might have downtime".

Computers aren't magic, everyone from heroku (you must have multiple dynos to be high availability) to ec2 (multiple instances across AZs) agree on "a single machine is not redundant". I don't see how fly's messaging is out of line with that. They don't tell you anywhere "Our apps and machines are literally magic and will never fail".

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#405

Post by Jupe »

wgjordan wrote: post #397
throwawaaarrgh wrote: post #338

There's a lot of bullshit in this HN thread, but here's the important takeaway: - it seems their staff were working on the issue before customers noticed it. - once paid support was emailed, it took many hours for them to respond. - it took about 20 hours for an update from them on the downed host. - they weren't updating their users that were affected about the downed host or ways to recover. - the status page was b…

(Fly.io employee here) To clarify, we communicated this incident to the personalized status page [1] of all affected customers within 30 minutes of this single host going down, and resolved the incident on the status page once it was resolved ~47h later. Here's the timeline (UTC): - 2023-07-17 16:19 - host goes down - 2023-07-17 16:49 - issue posted to personalized status page - 2023-07-19 15:00 - host is fixed - 202…

Ouch?

The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. For an entire cluster to be down for that long is just unacceptable. Rebuilding a cluster from the last-known-good backup should not take that long, unless there are PBs of data involved; dividing such large data stores into separate clusters/instances seems warranted. Solution archs should steer customers to multiple, smaller clusters (sharding) whenever possible. It is far better to have some customers impacted (or just some of your customer's customers) than have all impacted, in my not so humble opinion.

And, if the data size is smaller, you may want to trigger a full rebuild earlier in your DR workflows just as an insurance policy.

The good news is that only a single cluster was impacted. When the "big boys" go down, everything is impacted... but customers don't really care about that.

Not sure if this impacted customer had other instances that were working for them?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#406

Post by kunley »

tptacek wrote: post #379
seti0Cha wrote: post #372

Earlier quoted context omitted.

Not a great summary from my perspective. Here's what I got out of it: - Their free tier support depended on noticing message board activity and they didn't. - Those experiencing outages were seeing the result of deploying in a non-HA configuration. Opinions differ as to whether they were properly aware that they were in that state. - They had an unusually long outage for one particular server. - Those points combined…

Apologies for repeating myself, but: You get to a certain number of servers and the probability on any one day that some server somewhere is going to hiccup and bounce gets pretty high. That's what happened here: a single host in Sydney, one of many, had a problem. When we have an incident with a single host, we update a notification channel for people with instances on that host. They are a tiny sliver of all our us…

> somehow a containerd boltdb on that host got corrupted, so when the machine bounced, containerd refused to come back online. We use containerd as a cache

Hey, even if I can feel sympathetic for the course of unfortunate events, it's hard to not to comment:

if you're using a cache, you should invalidate it on failure!

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#407

Post by jim180 »

throwaway220033 wrote: post #203
jim180 wrote: post #196

Earlier quoted context omitted.

I'm using mrsk + DO instances (DB is running on managed instance within DO) on my side project, which is Django. I'm very happy about everything. No complexity, easy to deploy and setup.

How do you compare DO to Hetzner?

I've picked up DO just because they have managed database instances.

If hetzner had this, I would have picked them instead.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#408

Post by wofo »

pech0rin wrote: post #28

I really want to love Fly.io. It's super easy to get setup and use, but to be honest I don't think anyone should be building mission critical applications on their service. I ended up migrating everything over to AWS (which I reallllly didn't want to do) because: * Frequent machines not working, random outages, builds not working * Support wasn't responsive, didn't read my questions (kept asking same questions over a…

My experience has also been somewhat disappointing. I had a toy project that I decided to host elsewhere (Hetzner VM + Dokku), after the node for the PG database stopped working without any notification and didn't come back online (until I manually resurrected it).

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#409

Post by wgjordan »

CoolCold wrote: post #403
seti0Cha wrote: post #396

Earlier quoted context omitted.

It seems to me like there's room for improving your customers' awareness around what is required for HA and how to tell when they are affected by a hardware issue. On the other hand, it may just be that the confusion is mostly amongst the casual onlookers, in which case you have my sympathies!

I'm not sure on this, will it make any sense - customers who DON'T WANT to be aware of what is required for HA (say lonely devs) choosing such a hosting types. Even if you put educational articles, I'm unsure it will be used. Putting some BANNER IN RED LETTERS into CLI output + link to article may work, though. What do you think?

This is exactly how it currently works:

  $ fly volumes create mydata
  Warning! Individual volumes are pinned to individual hosts.
  You should create two or more volumes per application.
  You will have downtime if you only create one.
  Learn more at https://fly.io/docs/reference/volumes/
  ? Do you still want to use the volumes feature? (y/N)
(and yes, the warning is already even in red letters too)

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#410

Post by mrkurt »

Jupe wrote: post #405
wgjordan wrote: post #397

Earlier quoted context omitted.

(Fly.io employee here) To clarify, we communicated this incident to the personalized status page [1] of all affected customers within 30 minutes of this single host going down, and resolved the incident on the status page once it was resolved ~47h later. Here's the timeline (UTC): - 2023-07-17 16:19 - host goes down - 2023-07-17 16:49 - issue posted to personalized status page - 2023-07-19 15:00 - host is fixed - 202…

Ouch? The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. For an entire cluster to be down for that long is just unacceptable. Rebuilding a cluster from the last-known-good backup should not take that long, unless there are PBs of data involved; dividing such large data stores into separate clusters/instances seems warranted. Solution archs should steer custom…

This was a single physical server running multiple VMs using local NVMe storage. It impacted a small fraction of customers.
Post reply on HN