Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

71–80 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#71

Post by arcanemachiner »

danjac wrote: post #66
aledalgrande wrote: post #9

Wondering if for small/bootstrapped projects there's any alternative people suggest? Fly has a nice UX and accessible prices, but it's unstable at best. I use the big clouds at work, but for personal they are $$$. Also I want to keep devops tending asymptotically to zero.

I use Dokku on top of Hetzner for my hobby projects - hosting is super cheap, for a little extra I can add a mounted volume for storage, and if the project outgrows a single server I can always just break out of Dokku and use some Docker containers behind a load balancer. If you are outside of Europe, Digital Ocean or Linode may work better for you.

Hetzner has 2 data centers in the US now. 1 in the east and 1 in the west.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#72

Post by nerpderp82 »

pier25 wrote: post #46
mrcwinn wrote: post #18

Earlier quoted context omitted.

For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.

So you didn't have a HA setup with multiple machines and volumes?

Is that even possible on Fly?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#73

Post by yla92 »

spiderice wrote: post #14

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…

Is it me or the page is now gone?

"Oops! That page doesn’t exist or is private."

Edit: Ok, I can see after sign up / log in.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#74

Post by teraflop »

gowthamgts12 wrote: post #17
spiderice wrote: post #14

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…

> While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If it really got missed, then I don't understand how the thread was made private to only logged-in users?

It looks like all 166 threads with the "App not working" tag are invisible when not logged in. So I'm guessing somebody applied that tag retroactively.

https://community.fly.io/c/questions-and-help/app-not-workin...

EDIT: it now appears that the "app-not-working" tag itself has been deleted, and no longer shows up even when logged in.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#75

Post by CameronNemo »

unmole wrote: post #27
xyzzy_plugh wrote: post #7

I think fly.io is pretty incredible but I can't help but feeling they're doomed to follow in heroku's footsteps (unclear if good or bad). They've built some pretty wild stuff and I can't help but wonder if they're overcooking the ocean instead of just solving problems for their users. Durable and available storage are all they really need to draw me away from big cloud providers but this combined with their answer to…

> use S3 or run minio Is using Cloudflare R2 not an option?

Backblaze even has an s3 api these days.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#76

Post by xixixao »

aledalgrande wrote: post #42
sho wrote: post #40

Earlier quoted context omitted.

Honestly these days I am leaning towards this approach: https://github.com/mrsked/mrsk/ It's all just docker.

Nah I don't wanna be responsible for running a control plane. I just wanna focus on the app, that's all.

No devops, focus on writing your app: https://www.convex.dev/

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#77

Post by skullone »

justinclift wrote: post #57
skullone wrote: post #22

Earlier quoted context omitted.

What hardware are you buying? Across tens of thousands of physical nodes in my environment, only a few would have "fatal" enough problems that required manual intervention per year. Yes we had hundreds of drives die a year, some ECC ram would exceed error thresholds, but downtime on any given node was rare (aside from patching, but we'd just live migrate KVM instances around as needed.

Maybe there needs to be a better "burn in" test setup for their new hardware, just to catch mistakes in the build prep and/or catch bad hardware?

Not that nothing will fail - but some manufacturers have just really good fault management, monitoring, alerting, etc. And even the simplest shit like SNMP with a few custom MIBs from the vendor (which theres some that do it better). Facilities and vendors that lend a good hand with remote hands is also nice, if you remote management infrastructure should fail. But out of band, full featured management cards with all the trimmings work so well. Some do good Redfish BMC/JSON/API stuff too on top of the usual SNMP and other nice builtin Easy Buttons. And today's tooling with bare metal and KVM, working around faults to be quite seamless. Even good NVME raid options if you just absolutely must have your local box with mirrored data protection, 10/40/100Gbps cards with a good libvirt setup to migrates large VMs in mere minutes, resuming on the remote end with nigh 1ms blip.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#80

Post by tptacek »

Y'all, this is going to be deeply unsatisfying, but it's what I can report personally:

I have no earthly clue why this thread on our community site is unlisted.

We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread (the site is public, anybody can sign up for an account and see the thread).

Say what you want about us, but hiding out from stuff like this isn't one of our flaws. When I find out more about what happened with this thread, I'll let you know (or Kurt will reply here and tell me I'm wrong).

I don't know enough about what happened with this Sydney server to be helpful to people who had instances running on it. When I know more about it, I'll be helpful, but I'm just learning about this stuff right now, after getting back in from a night out.

Almost immediately afterwards

It looks like... all the posts in the app-not-working category are "private"? Like it's some setting on the category itself? "Private" here means you need to have signed up for a Discourse account to see them?

Post reply on HN