41–50 of 123 posts
"This impacts queries to our API, including creating and modifying apps, as well as incoming network requests for recently deployed apps." Would be really interested to understand why it affects recently deployed apps but not apps that are already established - something to do with how the Fly Router works?
> We are working to build a new Consul cluster with 10x the RAM. Oh boy. I wouldn’t wanna be the people doing this. Working with infrastructure is hard . Doing it under tight SLAs? Ugh. I really hope the people working on this are being well supported.
I attempted to deploy a simple app on Fly a couple of weeks ago, but porting it from heroku became a nightmare, servers crashing, cryptic error messages, etc. Maybe I'm in the minority but in any case my experience with Fly definitely left me questioning the hype around it.
Most others require pretty decent Docker knowledge.
> We are working to build a new Consul cluster with 10x the RAM. Oh boy. I wouldn’t wanna be the people doing this. Working with infrastructure is hard . Doing it under tight SLAs? Ugh. I really hope the people working on this are being well supported.
2. The SLA fly.io has commits to 99.9% uptime, meaning they can "afford" ~1.5m downtime daily, or ~40m monthly. AWS "offers" 99.99% (~4m monthly) if I recall correctly, but their scale is also wildly different obviously.
This has been a rough week, and I'm sorry we broke peoples' apps. We had a big Nomad outage on Monday, and then a suspiciously similar Consul outage today. Both tipped over faster than we could detect and mitigate, and we ended up having to do serious surgery to build entirely new Consul/Nomad clusters. There's nothing to brag about here, I just wanted to let y'all know we're listening (even when things aren't on the…
Shame cuz we were excited about our nomad+consul+vault setup and invested a lot of money into building it. But just didn’t have the time or enough depth of expertise to babysit it.
I have seen some issues around Consul these days. As a person with no background in distributed systems, I am wondering why people choose Consul over alternatives. Are there features that etcd doesn't offer?
I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.
I have seen some issues around Consul these days. As a person with no background in distributed systems, I am wondering why people choose Consul over alternatives. Are there features that etcd doesn't offer?
If you want apps to discover each other and be able to communicate effortlessly, even across datacenters, Consul, in theory, enables this.
I say in theory because I couldn't get federated Consul actually working.
Fly is building everything in hard mode - since they are not layering on top of an existing cloud like pretty much everyone else (heroku, render, railway, ...). It's either very smart (if they pull it off) because they will have a ginormous cost advantage or they fail. I'm personally of the opinion that the ux on top of aws/gcp/... is worse than a doo-doo in a shoe. However, they are as stable as can be (all complex…
Are they really building everything in hard mode or do they just have a bad architecture?
Fly is building everything in hard mode - since they are not layering on top of an existing cloud like pretty much everyone else (heroku, render, railway, ...). It's either very smart (if they pull it off) because they will have a ginormous cost advantage or they fail. I'm personally of the opinion that the ux on top of aws/gcp/... is worse than a doo-doo in a shoe. However, they are as stable as can be (all complex…
Are they really building everything in hard mode or do they just have a bad architecture?
Earlier quoted context omitted.
Same. I’m so disappointed because I’ve been rooting for them. We were close to a major deployment/migration (well, major as is mid four figures per month, not major like Google) but they were removed from the decision set. It would not have been responsible to bet on them at this time. I hope they get this sorted - they’re really good folks!
Thank you! I'm both sorry it didn't work out (because $$$$) and also glad we didn't create any agony for you. Someday, we hope to create mild irritation for you, though, if we can.
But these stability issues actually make me more nervous about the fact that I’d have to manage my own postgres cluster and have to learn how to recover it in such an event. AWS RDS has made me soft!
Wishing you guys the best. We’ll still use fly for QA until a few of these issues are sorted out. And until there’s fully managed pg (first party or third party)