Fly.io Status – Consul cluster outage
status.flyio.net
Fly.io Status – Consul cluster outage
1–10 of 123 posts
Re: Fly.io Status – Consul cluster outage
#2They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?
Re: Fly.io Status – Consul cluster outage
#3They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?
The latter (they openly admit as such)
Re: Fly.io Status – Consul cluster outage
#4Respect to anybody who is an SRE at fly.io. Couldn’t pay me enough to do that job
Re: Fly.io Status – Consul cluster outage
#5I really really wanted to like and recommend fly.io but I wouldn't risk deploying anything more than a side project to tinker with, given how many random issues I encountered in a relatively short development time. It was a simple Phoenix app which made me wonder "am I doing things totally wrong?" quite a few times, after exhausting all info sources.
But when I tried the same process the next day, it would deploy just fine. Plus the outages that appear to be getting more frequent don't make me optimistic.
At least they're transparent about their issues, gotta give them that. I still kinda root for them, maybe they'll make a comeback.
Re: Fly.io Status – Consul cluster outage
#6Relevant: "Reliability: It's not great" from last week https://news.ycombinator.com/item?id=35044516
They even specifically call out Consul as a source of trouble.
> We propagate app instance and health information across all our regions. That’s how our proxies know where to route requests, and how our DNS servers know what names to give out.
> We started out using HashiCorp Consul for this. But we were shoehorning Consul, which has a centralized server model design for individual data center deployments, into a global service discovery role it wasn’t suited for. The result: continuously stale data, a proxy that would route to old expired interfaces, and private DNS that would routinely have stale entries.
Re: Fly.io Status – Consul cluster outage
#7Respect to anybody who is an SRE at fly.io. Couldn’t pay me enough to do that job
They just hired their first if I recall correctly. I feel for their customers more than I do for their shareholders
Re: Fly.io Status – Consul cluster outage
#8They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?
The former ish -- they relied on consul marketing that the hammer fit the square hole. Hashicorp has been pretty bad about marketing themselves as the right tool for any job, but they really only fit the narrowest of tasks before you find yourself needing an alternative or being compelled to buy a support contact.
It's atlassian from Arkansas, just faster
Re: Fly.io Status – Consul cluster outage
#9Been a fan of fly and have had most, if not all, of my side and semi-side projects on there for some time now. But...the ratio of good/fun/snarky blog posts to reliable service has gotten a bit too large for me, starting to look for other providers at this point just in case they can't turn this trend around. Honestly been a good object lesson for me in the importance of backing up marketing/hype/"mind-share" stuff w/ absolute rock-solid performance/reliability or just forgoing the former for the latter.
As an aside, it's also taking down some decently-load-bearing web infra like unpkg => https://www.unpkg.com/
Re: Fly.io Status – Consul cluster outage
#10Respect to anybody who is an SRE at fly.io. Couldn’t pay me enough to do that job
Markdowns forged in god-steel coming out from this incident