I attempted to deploy a simple app on Fly a couple of weeks ago, but porting it from heroku became a nightmare, servers crashing, cryptic error messages, etc. Maybe I'm in the minority but in any case my experience with Fly definitely left me questioning the hype around it.
Fly.io Status – Consul cluster outage
21–30 of 123 posts
Re: Fly.io Status – Consul cluster outage
#22Interestingly, Roblox went down for 73 hours due to a "unique" issue with Consul as well [1].
Great read on how the issue was approached, handled, and ultimately remediated.
[1] https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
Re: Fly.io Status – Consul cluster outage
#23I really really wanted to like and recommend fly.io but I wouldn't risk deploying anything more than a side project to tinker with, given how many random issues I encountered in a relatively short development time. It was a simple Phoenix app which made me wonder "am I doing things totally wrong?" quite a few times, after exhausting all info sources. But when I tried the same process the next day, it would deploy jus…
Same. I’m so disappointed because I’ve been rooting for them. We were close to a major deployment/migration (well, major as is mid four figures per month, not major like Google) but they were removed from the decision set. It would not have been responsible to bet on them at this time. I hope they get this sorted - they’re really good folks!
Re: Fly.io Status – Consul cluster outage
#24That's the issue with centralized infra... I expect it to be less and less stable the more customers they have. I still wish them good luck.
On my side I took the opposite direction, each workload is shared nothing.
Re: Fly.io Status – Consul cluster outage
#25From my experience etcd would have been a better choice for maturity if they don't need the gossip stuff.
Re: Fly.io Status – Consul cluster outage
#26They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?
Roblox had a massive 3-day outage [1] in October 2021 due to a Consul feature that didn’t work as expected. My gut with Consul is don’t use it for high-load distributed services. [1] https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
The Roblox outage seemed like a pretty one-off instance due to a hard to catch bug. Consul still seems like a great choice and it looks like Roblox continues to use it at their scale.
Re: Fly.io Status – Consul cluster outage
#27Earlier quoted context omitted.
Relevant response from the Fly community forums: https://community.fly.io/t/frequent-outages-is-really-demons...
Yeah, I saw; I've kept up w/ everything pretty closely. Still decently frustrating as a paying customer, but I hope they can figure it out. If they can and can show some real reliability, I'll be an even bigger fan.
Yep! More putting it out there for other folks. I’m also a somewhat frustrated paying customer, but as I’m dealing with my own growing pains, I relate to what they’re going through. I’ve personally migrated my DB to Crunchy to somewhat mitigate the risk.
Re: Fly.io Status – Consul cluster outage
#28Interestingly, Roblox went down for 73 hours due to a "unique" issue with Consul as well [1]. Great read on how the issue was approached, handled, and ultimately remediated. [1] https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
Most often the issues that take down a site are with core services like network routing, DNS and service discovery. Consul gets mentioned because it’s in that business and isn’t a standard so it gets called out specifically. Zookeeper, HAProxy and various cluster managers also get slagged for this stuff and yeah, sometimes it’s their fault but that’s what it means to be in that business.
Re: Fly.io Status – Consul cluster outage
#29Earlier quoted context omitted.
Yeah, I saw; I've kept up w/ everything pretty closely. Still decently frustrating as a paying customer, but I hope they can figure it out. If they can and can show some real reliability, I'll be an even bigger fan.
Yep! More putting it out there for other folks. I’m also a somewhat frustrated paying customer, but as I’m dealing with my own growing pains, I relate to what they’re going through. I’ve personally migrated my DB to Crunchy to somewhat mitigate the risk.
That's a good point / thing that I've not thought about as much here. I would be much more frustrated if my primary datastores were hosted there. As things stand, their semi-hosted offering never really made sense to me (esp. now lol), but I do think if you get into the game of DB hosting there's almost another level of expectation even beyond basic compute (oddly enough)
Re: Fly.io Status – Consul cluster outage
#30Was affected by the outage. Didn't know about it so I thought it was just another crash on Fly.io.
Tried to restart our app from the command line, only to be told they had disabled the API. And there is no restart feature on their dashboard. So all I could do was watching flyio logs telling me that our apps were down.
Sigh.
We moved from Heroku to Fly.io only this January, and are already considering moving away from it. The reliability is miserable at best. And so many basic features are missing. Yes it's much cheaper than Heroku, but we ended up paying much more time/resource/money dealing with its glitches. Defeats the purpose why we used a PaaS in the first place.