Fly.io Postgres cluster down for 3 days, no word from them about it • Hacker News Board

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#131

Post by throwawayfly » Fri, Jul 21, 2023, 6:31 AM UTC

p-e-w wrote: post #68

gowthamgts12 wrote: post #17

Earlier quoted context omitted.

> While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If it really got missed, then I don't understand how the thread was made private to only logged-in users?

Whoa, what? That's a much bigger red flag than the downtime itself.

Ok as long as we’re getting conspiratorial, something similar I observed has bugged me.

About a year ago fly awarded a few people in the forums, I think it was 3, the “aeronaut” badge. Basically just pointless bling for a “routinely very helpful” person or somesuch. Still, I can imagine it was cool to get it. No, it wasn’t me.

One person I saw with it absolutely deserved it: this person is, to this day, always hopping in and helping people; linking to docs; raising their own issues with a big dose of “fellow builder” understanding and empathy; that sort of person. My own queries typically led me to a thread that this person has answered. In short - the kind of helpful, proactive, high knowledge volunteer early adopter that every community needs - and a handful are blessed to find.

Then one day I saw this same person had offered — to one random newbie with build problems in one of the many HALP threads — a reply like, “maybe Fly isn’t the best option for you. here are some other places that can host an app”.

The thread was left alone and faded, like many when a lost newbie is involved. But 1 day later, I noticed this tireless early adopter no longer had their “aeronaut” badge.

I still refuse to believe my own eyes about something that petty.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#132

Post by wg0 » Fri, Jul 21, 2023, 6:32 AM UTC

Not a sarcastic or rhetorical question - how come the three big A clouds or even smaller ones (Hetzner,my favorite) are mostly so stable (give or take some outages) and anyone knows their internal engineering, architecture and practices to keep systems that much stable?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#133

Post by skullone » Fri, Jul 21, 2023, 6:33 AM UTC

justinclift wrote: post #117

skullone wrote: post #77

Earlier quoted context omitted.

Not that nothing will fail - but some manufacturers have just really good fault management, monitoring, alerting, etc. And even the simplest shit like SNMP with a few custom MIBs from the vendor (which theres some that do it better). Facilities and vendors that lend a good hand with remote hands is also nice, if you remote management infrastructure should fail. But out of band, full featured management cards with all…

Good point. :) I'm still wondering about their hardware acceptance/qualification though, prior to it being deployed. ;)

Yah presumably they put stuff through it's paces and give everything good fit and finish before running workloads. But failures do happen either way

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#134

Post by subarctic » Fri, Jul 21, 2023, 6:36 AM UTC

bkazez wrote: post #78

This is why people should just run a small managed k8s cluster on GCP. This has worked super well for us. https://www.vmii.org/blog/2023/03/12/kubernetes/

I tried using kubernetes a while back for hosting a side project on a raspberry pi. I guess technically I was running microk8s on the pi and had to install kubectl locally to interact with it.

I actually like some of the concepts, like pods and ingress, but one thing I noticed that I didn't like, as far as I remember, was that there's not really a good way in kubernetes to make your YAML more dynamic. Apparently you're supposed to use these other things like Helm Charts, which isn't even part of kubernetes?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#135

Post by nerpderp82 » Fri, Jul 21, 2023, 6:36 AM UTC

fmajid wrote: post #118

nerpderp82 wrote: post #72

Earlier quoted context omitted.

Is that even possible on Fly?

He may have been talking about Fly themselves. Certainly having only a single machine to serve a wealthy metropolis of 8 million people seems like amateur hour.

Fly sounds like they need some Conway's Law. A front end that designs the nice api and works on developer affordances and the backend that keeps it running and reliable.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#136

Post by brylie » Fri, Jul 21, 2023, 6:37 AM UTC

NicoJuicy wrote: post #91

steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I moved from DO to Hetzner ( cheaper), I am happy about it.

I'm enjoying the DO App Platform (Heroku alternative). Do you know if Hetzner has a similar service that I could compare?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#137

Post by system2 » Fri, Jul 21, 2023, 6:38 AM UTC

wg0 wrote: post #132

Not a sarcastic or rhetorical question - how come the three big A clouds or even smaller ones (Hetzner,my favorite) are mostly so stable (give or take some outages) and anyone knows their internal engineering, architecture and practices to keep systems that much stable?

IMHO it is their approach. I use Hetzner and OVH (and their other variants for lower budget clients) for our EU clients. They do not use buzz words like "deploy app server", "cloud clusters", "turbo charge this app". They are simply providing VPS and similarly configured droplets. They are also established and don't want to mess around with very modern experimental infrastructures.

Same goes for Digital Ocean. No buzz words. Just hosting with droplets. They simply say "here pick a linux distro, configure whatever and don't ask us much about app support". I use their Linux distros for my own apps and if want anything extra I just install it and suffer my own actions' consequences. Not theirs.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#138

Post by TekMol » Fri, Jul 21, 2023, 6:42 AM UTC

What do people get out of using special services like Fly.io instead of standard VMs like the ones you can get from $5/month these days?

Can anybody who uses Fly.io explain their rationale? Why do the additional integration with Fly.io, trust and install their special software on your machines and tie your project into their ecosystem?

What type of application are you running? How many users are using it?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#139

Post by quickthrower2 » Fri, Jul 21, 2023, 6:43 AM UTC

mrcwinn wrote: post #18

spiderice wrote: post #14

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…

For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.

Fly is in my “try later book” from a year or two ago. I remember it was hard to deploy anything due to downtime so gave up. Sad that stuff like this still happens.

You shouldn’t need to multi region a postgres yourself - they should have at least 2 data centre redundancy for the region and it just works.

Hope they get some magic sauce to become better at this.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#140

Post by realusername » Fri, Jul 21, 2023, 6:44 AM UTC

brylie wrote: post #136

NicoJuicy wrote: post #91

Earlier quoted context omitted.

I moved from DO to Hetzner ( cheaper), I am happy about it.

I'm enjoying the DO App Platform (Heroku alternative). Do you know if Hetzner has a similar service that I could compare?

Personally I just install Dokku onto the machine, it replaced all my Heroku (and competitors) uses.

Additionally, you still keep the full ssh access to the machine if you ever need it.