Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

31–40 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#31

Post by steve_adams_86 »

mrcwinn wrote: post #18
spiderice wrote: post #14

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…

For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited.

I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers.

I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments go missing with no automatic recovery. I didn’t realize this was happening to a number of my services until I started monitoring with 3rd party tools, and it became evident that I really couldn’t rely on them.

It’s a shame because I do like a lot of other things about them. Even for hobby work it didn’t seem worth the trouble. With digitalocean, everything “just works”. There’s no free tier, but the lower end of pricing means I can run several Go apps off of the same droplet for less than the price of a latte. It’s worth the sanity.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#32

Post by steve_adams_86 »

pech0rin wrote: post #28

I really want to love Fly.io. It's super easy to get setup and use, but to be honest I don't think anyone should be building mission critical applications on their service. I ended up migrating everything over to AWS (which I reallllly didn't want to do) because: * Frequent machines not working, random outages, builds not working * Support wasn't responsive, didn't read my questions (kept asking same questions over a…

My experience was the same. I stopped using it for hobby projects recently when I had two consecutive days of being unable to build anything. The same stuff that built the week before, built fine locally, then eventually built on fly again — just, inexplicable downtime with no word from support.

Their free tier is very generous. You can get a lot happening and stay under their billing threshold. But, I like to get stuff done. I have a family. I code in my spare time very rarely, and I need a service that’ll let me just build my goddamn project. This was a small static site built by Node, so nothing spectacular happening.

I do wish them the best though. They have an excellent product in their tooling, and if they could stabilize their infrastructure I’d love to try them again.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#33

Post by yowlingcat »

xyzzy_plugh wrote: post #7

I think fly.io is pretty incredible but I can't help but feeling they're doomed to follow in heroku's footsteps (unclear if good or bad). They've built some pretty wild stuff and I can't help but wonder if they're overcooking the ocean instead of just solving problems for their users. Durable and available storage are all they really need to draw me away from big cloud providers but this combined with their answer to…

At least I could rely on Heroku in production. I've wanted to give Fly.io a try but this gives me pause. I really do miss the Heroku DX whenever I'm putzing around with the increasing complexity of AWS.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#34

Post by q7xvh97o2pDhNrh »

aledalgrande wrote: post #9

Wondering if for small/bootstrapped projects there's any alternative people suggest? Fly has a nice UX and accessible prices, but it's unstable at best. I use the big clouds at work, but for personal they are $$$. Also I want to keep devops tending asymptotically to zero.

Maybe just pick up 3 chonky EC2 boxes, set up iptables on each of them, have each one run a containerized version of your code that gets built and deployed from CI every time you push to Github, slap an ALB in front of it all, and call it a day?

And if you need state, then spin up a little RDS with your favorite SQL flavor of choice?

The CI deploy script could even bake in little health-checks so you can do rolling deploys with zero downtime. Depending on how fancy you wanted to get with your shell scripting, you could probably even make 1 of your 3 boxes a canary without too much trouble.

I'm realizing I haven't thought about this in a long time, since nowadays I just get to use the fancy stuff at work. Kind of a fun thought experiment!

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#35

Post by emmelaich »

spiderice wrote: post #14

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…

The irony or perhaps the tragedy of building a low friction service is that you have to have experts on the lower level high friction stuff.

I would hope that after a couple of hours downtime, they'd bring up a fresh machine with Ansible or whatever. Hardware or AWS/GCP Vm.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#36

Post by mike_d »

DumbStarbucks wrote: post #3

You unfortunately get what you pay for. AWS is more expensive than God, but I'll be damned if you can't have a throat to choke in less than 10 minutes whenever something like this happens.

> I'll be damned if you can't have a throat to choke in less than 10 minutes whenever something like this happens

That is a hell of generous description for a person who sits in your Slack instance and responds with "I have escalated to the team internally and am waiting to hear back on confirmation if this is an issue."

Moving a Level 1 support engineer closer to the customer doesn't give them more information, it just reduces the latency to getting a non-answer.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#37

Post by sho »

orangepurple wrote: post #10
DumbStarbucks wrote: post #3

You unfortunately get what you pay for. AWS is more expensive than God, but I'll be damned if you can't have a throat to choke in less than 10 minutes whenever something like this happens.

AWS support replies back to your messages when they feel like it. Their support is just as shady but they have better uptime for sure

No love for AWS, but this isn't true, at least for larger deploys. If you're running enough with them that you have an account manager, they are very good indeed. You can have someone, someone good, on the phone within minutes and they will stay on the line until the issue is sorted.

I recall an incident at my old company where we were under DDOS, it was getting through cloudflare and saturating LBs in some complicated manner (don't recall the exact details) which made it hard for us to fix ourselves. They were on the phone with us for hours, well past midnight their time, helping us sort it out. The downtime sucked, but I was certainly impressed with their truly excellent support.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#39

Post by iamyatin »

reustle wrote: post #19
aledalgrande wrote: post #9

Wondering if for small/bootstrapped projects there's any alternative people suggest? Fly has a nice UX and accessible prices, but it's unstable at best. I use the big clouds at work, but for personal they are $$$. Also I want to keep devops tending asymptotically to zero.

I’m quite happy with https://render.com after leaving Heroku

I second render.com. I switched from fly.io to Render.com after seeing a few of my instances getting bottlenecked and crashing. Now the same service runs smoothly on render.com without any crashes. Didn't dig any deeper but somehow the resource management is better with render.com

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#40

Post by sho »

aledalgrande wrote: post #9

Wondering if for small/bootstrapped projects there's any alternative people suggest? Fly has a nice UX and accessible prices, but it's unstable at best. I use the big clouds at work, but for personal they are $$$. Also I want to keep devops tending asymptotically to zero.

Honestly these days I am leaning towards this approach: https://github.com/mrsked/mrsk/

It's all just docker.

Post reply on HN