Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Status – Consul cluster outage

status.flyio.net

Post reply on HN
Print view View original

31–40 of 123 posts

Re: Fly.io Status – Consul cluster outage

#31

Post by mrkurt »

felixding wrote: post #30

Was affected by the outage. Didn't know about it so I thought it was just another crash on Fly.io. Tried to restart our app from the command line, only to be told they had disabled the API. And there is no restart feature on their dashboard. So all I could do was watching flyio logs telling me that our apps were down. Sigh. We moved from Heroku to Fly.io only this January, and are already considering moving away from…

I know blocking deploys sucks, I'm sorry. We disabled them to prevent otherwise healthy apps from going down. When Consul fails, we can't boot new app processes. The ones that are already running continue running. A restart is roughly the same as a deploy, in this respect.

Re: Fly.io Status – Consul cluster outage

#32

Post by mrkurt »

This has been a rough week, and I'm sorry we broke peoples' apps. We had a big Nomad outage on Monday, and then a suspiciously similar Consul outage today. Both tipped over faster than we could detect and mitigate, and we ended up having to do serious surgery to build entirely new Consul/Nomad clusters.

There's nothing to brag about here, I just wanted to let y'all know we're listening (even when things aren't on the HN front page).

Re: Fly.io Status – Consul cluster outage

#33

Post by pm90 »

> We are working to build a new Consul cluster with 10x the RAM.

Oh boy. I wouldn’t wanna be the people doing this. Working with infrastructure is hard. Doing it under tight SLAs? Ugh. I really hope the people working on this are being well supported.

Re: Fly.io Status – Consul cluster outage

#34

Post by simonw »

"This impacts queries to our API, including creating and modifying apps, as well as incoming network requests for recently deployed apps."

Would be really interested to understand why it affects recently deployed apps but not apps that are already established - something to do with how the Fly Router works?

Re: Fly.io Status – Consul cluster outage

#35

Post by mrkurt »

abofh wrote: post #7
HL33tibCe7 wrote: post #4

Respect to anybody who is an SRE at fly.io. Couldn’t pay me enough to do that job

They just hired their first if I recall correctly. I feel for their customers more than I do for their shareholders

We've scaled infra ops from 3 to 7 people in the past few weeks. Our very first VP was a VP Infra Ops, because that's the thing we have to get best at to succeed as a business.

Note that we grew the whole company from 25 to 60 over the last six months.

Re: Fly.io Status – Consul cluster outage

#37

Post by mrkurt »

Thaxll wrote: post #2

They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?

The design of Consul is wrong for what we need to do. Consul has been pretty good when it's running, but it's a huge pain in the ass to recover when it falls over. And when it does fall over, it's usually with no notice.

Re: Fly.io Status – Consul cluster outage

#38

Post by mrkurt »

mrcwinn wrote: post #23
pawelduda wrote: post #5

I really really wanted to like and recommend fly.io but I wouldn't risk deploying anything more than a side project to tinker with, given how many random issues I encountered in a relatively short development time. It was a simple Phoenix app which made me wonder "am I doing things totally wrong?" quite a few times, after exhausting all info sources. But when I tried the same process the next day, it would deploy jus…

Same. I’m so disappointed because I’ve been rooting for them. We were close to a major deployment/migration (well, major as is mid four figures per month, not major like Google) but they were removed from the decision set. It would not have been responsible to bet on them at this time. I hope they get this sorted - they’re really good folks!

Thank you! I'm both sorry it didn't work out (because $$$$) and also glad we didn't create any agony for you. Someday, we hope to create mild irritation for you, though, if we can.

Re: Fly.io Status – Consul cluster outage

#39

Post by candiddevmike »

suryao wrote: post #18

Fly is building everything in hard mode - since they are not layering on top of an existing cloud like pretty much everyone else (heroku, render, railway, ...). It's either very smart (if they pull it off) because they will have a ginormous cost advantage or they fail. I'm personally of the opinion that the ux on top of aws/gcp/... is worse than a doo-doo in a shoe. However, they are as stable as can be (all complex…

Are they really building everything in hard mode or do they just have a bad architecture?

Re: Fly.io Status – Consul cluster outage

#40

Post by mrkurt »

simonw wrote: post #34

"This impacts queries to our API, including creating and modifying apps, as well as incoming network requests for recently deployed apps." Would be really interested to understand why it affects recently deployed apps but not apps that are already established - something to do with how the Fly Router works?

We still pipe service discovery through Consul, we just propagate it with a different, gossip based mechanism. Services are stored in local sqlite DBs on every host that runs our Proxy. They are designed to keep running, even when we can't get updates to them.

This outage prevented us from writing services to Consul, so we couldn't read them back out. Nomad will only really write service information to Consul, so we're kind of stuck with Consul in the loop until we're fully off Nomad.

Post reply on HN