Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

351–360 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#351

Post by api »

throwawaymaths wrote: post #226
quickthrower2 wrote: post #139

Earlier quoted context omitted.

Fly is in my “try later book” from a year or two ago. I remember it was hard to deploy anything due to downtime so gave up. Sad that stuff like this still happens. You shouldn’t need to multi region a postgres yourself - they should have at least 2 data centre redundancy for the region and it just works. Hope they get some magic sauce to become better at this.

> Hope they get some magic sauce to become better at this. When I saw them describe their multiregion SQL replication architecture I thought "what crazy person thought this wouldn't eventually open up a spider's nest of distributed systems errors?"

CockroachDB does this, but that's the result of over 10 years of heads down hard-ass engineering and it's still slower than Postgres because distributed sync is not free. That means you have to provision it properly and with enough resources.

Their license would require a company like fly.io to pay them though, so I'm sure this resulted in fly.io instead trying to whip up an improvised infrastructure on the back of stock Postgres. I bet this cost them a whole lot more than paying CockroachDB would have, but devs have been conditioned that you should never ever pay for software even if it's the result of tons of deep engineering and solves massive brutal problems for you. I also bet there's some not-invented-here ego involved.

P.S. I don't work for CDB but I would absolutely consider them and we may end up using them at some point. They let you do a ton for free. They only charge for stuff you need if you get really really huge or if you are running a SaaS reselling DB services like fly.io would have been doing.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#352

Post by eropple »

silisili wrote: post #70
eropple wrote: post #51

Earlier quoted context omitted.

I was working for a pretty big early AWS customer--one that had realized that for the low low price of all your money you could make DynamoDB scale to some truly massive numbers--and one time when we were having trouble around noon Eastern, a colleague called up our TAM. As he told it, the TAM sounded half-asleep, so my colleague asked if everything was alright. "I'm in Hawaii on my honeymoon and my backup missed you…

Weird, I just begrudgingly went from Postgres to Dynamo because it was so much cheaper. We're not huge scale though, so I'm wondering where the costs start to diverge the other way.

This was, 2012 and we were hitting read and write limits regionally.

It was not a wise plan. It did, however, run. Technically.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#353

Post by fdsafsaf »

Used to love fly, then had a few issues.* The CEO wrote back in March they were working on reliability, but then you have this case study on what not to do in an incident response. 1) Fail to monitor your primary support channel. 2) Allow your support channel to become "private." 3) Not update your status page.

* First was some sort of certificate issue that cost me literally days of debugging that turned out to be their fault. * Then weirdness around their v2 deployments where I just can't grok some of the documentation.

Just use AWS. Your time is more valuable then what you're saving on the fly.io free plan.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#354

Post by graypegg »

steve_adams_86 wrote: post #31
mrcwinn wrote: post #18

Earlier quoted context omitted.

For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

DO actually does have a free tier! If you use their “app platform” (their equivalent to fly/heroku/render/etc) you can host 3 “static” apps for free. So if you have a Hugo/Jekyll blog or something, it’ll set up a whole little CD system for it for free.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#355

Post by bjord »

danielvaughn wrote: post #56
steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

historically, I've used Vultr, but I don't see anyone talking about it—I'm curious if anyone else has thoughts on them? (I've been happy, but then again my usage has been exceedingly basic)

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#356

Post by plagiarist »

spiderice wrote: post #14

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…

Why is it my responsibility to move instances from machine to machine to mitigate a cloud host's outages? What is their utility if not performing the bare minimum of cloud host responsibilities keeping my container up?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#357

Post by Ensorceled »

TekMol wrote: post #198
danjac wrote: post #182

Earlier quoted context omitted.

There's a sweet spot of early startup or side project where you don't have the time, budget or people to manually set up and maintain servers on your own or deal with the complexity and cost of Kubernetes or AWS, especially when your focus is on building the product and acquiring customers. Heroku (before its inevitable enshittification under Salesforce) was great for this use case. Sure you will outgrow it at some p…

What exactly does Fly.io give you? You already know how to set up your project locally. Why not just do the same setup on any cloud VM and boom it is online?

Who fully sets up a significant project locally?

I used Heroku for a project mostly because my team didn't have skill set to set this up and I wasn't going to do it. As far as I know they are still on Heroku (with a smattering of AWS services) for that same reason: just works and cheaper than doing it yourself.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#358

Post by api »

smallerfish wrote: post #245
benjaminwootton wrote: post #184

Earlier quoted context omitted.

Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Even if customers are only running one instance, I would expect the whole thing to rebalance in an automated way especially with fly.io being so container centric. It also sounds like this is some managed Postgres service rather than users running only one instance of their container, so it’s even more reasonable to expect…

If you lose a single instance on RDS and you don't have replication set up, you'll also have downtime. (Maybe not with Aurora?) And +1 to the sibling comment; Fly makes it very clear that single instance postgres isn't HA, and talks about what you need to do architecturally to maintain uptime.

Yeah but you won't lose your data. They have backup infrastructure and EBS is rock solid.

Down time is one thing. Data loss is something else.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#359

Post by pc86 »

fauigerzigerk wrote: post #217
JanSt wrote: post #124

Earlier quoted context omitted.

I'm using Digital Ocean App platform, which does pretty much everything for me. It's very simple to use. I can run my app as a single developer without caring about infrastructure for 99% of the time.

Do they offer authentication/authorization? This is the one thing I need in every app and don't want to do myself.

In addition to Supabase Auth the sibling mentions (which I played with very briefly) I've been using clerk.dev (no affiliation) and it's great. Depending on your definition of doing it yourself it could be just want you want. You have to set some things up, you're not going to get things like row-level permissions you get out of the box w/ Supabase, but if you're looking for a quick implementation where things like password reset etc. are handled for you, it might be a good fit.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#360

Post by tedchs »

bjord wrote: post #355
danielvaughn wrote: post #56

Earlier quoted context omitted.

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

historically, I've used Vultr, but I don't see anyone talking about it—I'm curious if anyone else has thoughts on them? (I've been happy, but then again my usage has been exceedingly basic)

I've used Vultr for several years (hobby projects) with no issues. My favorite feature is having a BGP session from my VM, which is unusual among cloud providers. I have an AS and am able to advertise my own IPs from multiple Vultr instances (anycast).
Post reply on HN