They're less humble in communicating other things https://fly.io/blog/we-raised-a-bunch-of-money/
I could not agree more. When I read this my immediate thought was — all that money, and none spent on product marketing or copywriting. Oof.
Fly.io Postgres cluster down for 3 days, no word from them about it
11–20 of 493 posts
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#12Wondering if for small/bootstrapped projects there's any alternative people suggest? Fly has a nice UX and accessible prices, but it's unstable at best. I use the big clouds at work, but for personal they are $$$. Also I want to keep devops tending asymptotically to zero.
Although, i have never used them, you can explore railway.app. it is the closest to fly.io and never heard any bad things.
I personally at the moment use digitalocean without any issues, but there's always the maintenance overhead of managing a server yourself.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#13I like fly.io a lot and I want them to succeed. They're doing challenging work...things break. Have to admit it's disappointing to hear about the lack of communication from them, especially when it's something the CEO specifically called out that they wanted to fix in his big reliability post to the community back in March. https://community.fly.io/t/reliability-its-not-great/11253#s...
Yes, this. It's tough when you've already played your "we messed up but we're making it right" card, and then you continue to not have it right.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#14There is now a response to the support thread from Fly[1]:
> Hi Folks,
> Just wanted to provide some more details on what happened here, both with the thread and the host issue.
> The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If we’d seen it earlier, we’d have offered more details the.
> More on what happened: We had a single host in the syd region go down, hard, with multiple issues. In short, the host required a restart, then refused to come back online cleanly. Once back online, it refused to connect with our service discovery system. Ultimately it required a significant amount of manual work to recover.
> Apps running multiple instances would have seen the instance on this host go unreachable, but other instances would have remained up and new instances could be added. Single instance apps on this host were unreachable for the duration of the outage. We strongly recommend running multiple instances to mitigate the impact of single-host failures like this.
> The main status page (status.fly.io) is used for global and regional outages. For single host issues like this one we post alerts on the status tab in the dashboard (the emergency maintenance message @south-paw posted). This was an abnormally long single-host failure and we’re reassessing how these longer-lasting single-host outages are communicated.
> It sucks to feel ignored when you’re having issues, even when it’s not intentional. Sorry we didn’t catch this thread sooner.
[1] https://community.fly.io/t/service-interruption-cant-destroy...
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#15I like fly.io a lot and I want them to succeed. They're doing challenging work...things break. Have to admit it's disappointing to hear about the lack of communication from them, especially when it's something the CEO specifically called out that they wanted to fix in his big reliability post to the community back in March. https://community.fly.io/t/reliability-its-not-great/11253#s...
I appreciate the honest feedback. We could have done better communicating about the problem. We've been marking single host failures in the dashboard for affected users and using our status page to reflect things like platform and regional issues, but there's clearly a spot in the middle where the status we're communicating and actual user experience don't line up.
We've been adding a ton more hardware lately to stay ahead of capacity issues and as you would expect this means the volume of hardware-shaped failures has increased even though the overall failure probability has decreased. There's more we can do to help users avoid these issues, there's more we can do to speed up recovery, and there's more we can do to let you know when you're impacted.
All this feedback matters. We hear it even when we drop the ball communicating.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#16You unfortunately get what you pay for. AWS is more expensive than God, but I'll be damned if you can't have a throat to choke in less than 10 minutes whenever something like this happens.
AWS support replies back to your messages when they feel like it. Their support is just as shady but they have better uptime for sure
FWIW, our aws enterprise support reps are available 24/7 and usually respond within a few minutes.
But again, you get what you pay for.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#17There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…
> While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally.
If it really got missed, then I don't understand how the thread was made private to only logged-in users?
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#18There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…
For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#19Wondering if for small/bootstrapped projects there's any alternative people suggest? Fly has a nice UX and accessible prices, but it's unstable at best. I use the big clouds at work, but for personal they are $$$. Also I want to keep devops tending asymptotically to zero.
I’m quite happy with https://render.com after leaving Heroku
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#20Wondering if for small/bootstrapped projects there's any alternative people suggest? Fly has a nice UX and accessible prices, but it's unstable at best. I use the big clouds at work, but for personal they are $$$. Also I want to keep devops tending asymptotically to zero.
Although, i have never used them, you can explore railway.app. it is the closest to fly.io and never heard any bad things. I personally at the moment use digitalocean without any issues, but there's always the maintenance overhead of managing a server yourself.
I wish digitalocean offered decent pricing for spaces (s3). Unfortunately it starts at 5$, which is an enormous price for storing 70 small images, but s3 would greatly simplify my server management moving state entirely outside the server (managed database + managed object storage)