Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

321–330 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#321

Post by capableweb »

benjaminwootton wrote: post #184
spiderice wrote: post #14

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…

Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Even if customers are only running one instance, I would expect the whole thing to rebalance in an automated way especially with fly.io being so container centric. It also sounds like this is some managed Postgres service rather than users running only one instance of their container, so it’s even more reasonable to expect…

> Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life.

Depends on where in your development cycle you are. If you just got started and haven't even figured out what you're actually building (prototyping), you shouldn't really use a hosting provider that randomly lose instances.

If you're on the other hand have done everything to improve your applications performance, had to resolve through-output issues with a distributed architecture and now running 10+ instances, then losing one host shouldn't impact you too much. But you really shouldn't start this way, it's doing web services the hard way and introduces a lot of complexity you shouldn't want to deal with when you're still trying to find product market fit.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#323

Post by dangoodmanUT »

NicoJuicy wrote: post #91
steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I moved from DO to Hetzner ( cheaper), I am happy about it.

Hetzner has a record for going silent with issues FYI, just hit their reddit to see all the horror stories

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#324

Post by riwsky »

MuffinFlavored wrote: post #319
oefrha wrote: post #296

Earlier quoted context omitted.

I was confused why support for platform failure relies on a forum where employees may or may not check. After checking docs[1], apparently you have to be on a paid plan (at least $29/mo) to access email support, so you may not have it even you’re paying for resources. I won’t be using it for side projects where I’m okay with paying $5-10/mo but don’t want to have three day outages. [1] https://fly.io/docs/about/suppo…

Forewarning: I am not being critical of fly.io nor their free support whatsoever when I say this. From a technical perspective, could they have "been better" from a technical perspective? I see their name a lot on HN so I know they are doing really cool + advanced things and this is probably some super small edge case that slipped through the cracks. Could they have added some message / do we as the HN community feel…

They broke uptime for the paid tier, not just the free tier.

The relevance of paid/free is that free (and cheap paid) plans don’t get fly support over email

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#325

Post by linuxftw »

pritambarhate wrote: post #160

Here the even bigger red flag is that Fly doesn't have a (automated?) way to quickly move workload from a faulty server to a good server. Especially when containers (and orchestrators) have abstracted away the concept of data volumes which can be attached and detached. (Yes, it needs a lot of serious technical investment to provide this and I think it's one of the reasons storage is expensive on the big 3 clouds.) If…

They don't have remote attached storage, it's all local on the node, lvm based volumes. The data persistence is 24hr or manually created lvm snapshots that are exported to s3.

It's really not a place to run persistent workloads. If you run postgres there, you need to be prepared to either hot load your data into a new instance, or restore from backups.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#326

Post by pmarreck »

pinkcan wrote: post #145
pmarreck wrote: post #38

Reliability is everything. Why aren’t they monitoring their own machines (real or virtual) and getting fire alarms when there’s an outage?

monitoring means you might get called-in on your night-out who wants that?

Every web dev who is worth their salt knows what s/he/they signed up for.

For the unique privilege of being able to build machines out of thin air, I will accept the occasional weekend page

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#327

Post by dbingham »

danielvaughn wrote: post #56
steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

I love DO for projects where I don't need control. For my side project, I eventually migrated to AWS after running into a lot of issues with DO.

Things like they don't give you the postgres root user on their managed postgres. And I ran into issues trying to capture the deployments in code. Their terraform providers are pretty good, but still leave something to be desired. For all its many warts, I'm much happier back on AWS. It did end up more expensive, but it's worth it for the fine grained control in my case.

But I spent the last 5 years as a DevOps/SRE, so... uh... I'm picky.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#328

Post by api »

jdkfoo wrote: post #29
mrcwinn wrote: post #13

Earlier quoted context omitted.

Yes, this. It's tough when you've already played your "we messed up but we're making it right" card, and then you continue to not have it right.

Hosting service that cannot get basics right after a decade plus of solving these problems as an industry. Are we even trying or just repeating ourselves because we don’t know what else to do? How can the entire industry keep making the same basic errors? “Let’s keep it simp… ohh nope we invented a Turing complete language and customer service is terri… wait do we have customer service?” I get the world turning again…

I just got gigabit bidirectional fiber at home and honestly if I were doing personal stuff or doing very early bootstrapping I'd just host from here with a good UPS. No it wouldn't be data center reliability but it'd work at least until it was ready to put in something more resilient.

You can pay for a business class fiber link too. It's about twice as expensive but they have guaranteed outage response times which is really what you pay for.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#329

Post by riwsky »

capableweb wrote: post #321
benjaminwootton wrote: post #184

Earlier quoted context omitted.

Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Even if customers are only running one instance, I would expect the whole thing to rebalance in an automated way especially with fly.io being so container centric. It also sounds like this is some managed Postgres service rather than users running only one instance of their container, so it’s even more reasonable to expect…

> Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Depends on where in your development cycle you are. If you just got started and haven't even figured out what you're actually building (prototyping), you shouldn't really use a hosting provider that randomly lose instances. If you're on the other hand have done everything to improve your applications performance, had to…

GP is referring to fly.io architecting for single instance failures, not its customers.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#330

Post by ericpauley »

Holy hell, there are some hostile comments here!

I've had service issues on Fly that I've escalated to support in the past, and given my experience it feels highly unlikely that they tried sweep this under the rug or somesuch.

At the time we had deployed a small business workload (few 100$/mo in billings) and paid for their $29 support plan, so grain of salt there. We faced service issues and, while the service reliability did eventually push us to migrate, support was top-notch the whole way through. Support was happy to escalate as needed to try to help get a solution, with MrKurt eventually joining in and helping identify root causes. During the entire episode everyone was realistic about where issues could be (i.e., were open to the possibility of it being a Fly issue). As people from Fly have noted, they've historically been quite open about when they weren't the best choice.

Again, while service reliability has been an issue (and Fly has admitted this in the past and is working on it), I think the assumption of badfaith in this thread is pretty unprofessional. It's also a lesson in how hesitant people are to pay for support. $29 for access to a human is not a bad deal; we certainly got good value out of it.

Post reply on HN