Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Status – Consul cluster outage

status.flyio.net

Post reply on HN
Print view View original

51–60 of 123 posts

Re: Fly.io Status – Consul cluster outage

#51

Post by [deleted] »

mrkurt wrote: post #42
pm90 wrote: post #33

> We are working to build a new Consul cluster with 10x the RAM. Oh boy. I wouldn’t wanna be the people doing this. Working with infrastructure is hard . Doing it under tight SLAs? Ugh. I really hope the people working on this are being well supported.

You wouldn't necessarily know this from the outside, but we have _exceptional_ internal support when things go sideways. This is relatively new, up until about two months ago most incidents were run by 1.5 people. We had 7 people working this one today.

[deleted]

Re: Fly.io Status – Consul cluster outage

#52

Post by Mizza »

atonse wrote: post #45
mrkurt wrote: post #32

This has been a rough week, and I'm sorry we broke peoples' apps. We had a big Nomad outage on Monday, and then a suspiciously similar Consul outage today. Both tipped over faster than we could detect and mitigate, and we ended up having to do serious surgery to build entirely new Consul/Nomad clusters. There's nothing to brag about here, I just wanted to let y'all know we're listening (even when things aren't on the…

We had mysterious consul outages (and related nomad outages) causing us to never deploy our new hashicorp stack to production. Shame cuz we were excited about our nomad+consul+vault setup and invested a lot of money into building it. But just didn’t have the time or enough depth of expertise to babysit it.

From my experience with the Hashi stack, I don't think it's a coincidence that Fly has a lot of downtime and are a major Hashi user. Terraform makes excellent bait though.

Still love using Fly, please add static assets hosting/CDN.

Re: Fly.io Status – Consul cluster outage

#53

Post by pcthrowaway »

HL33tibCe7 wrote: post #3
Thaxll wrote: post #2

They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?

The latter (they openly admit as such)

They say that, but they're also being actively supported by Hashicorp right now (one would presume), so they really need to maintain a good working relationship.

I don't have a relationship with Hashicorp, and have tried using Consul. Everything about it is amazing in theory, but you might need a few years of experience with kube, consul, go, and maybe even the hashicorp stack to even begin debugging when things don't work as advertised.

I still think my company is going to take another stab at consul in the future, because we do need service discovery. But they're advertising a solution to an incredibly hard problem with a shit ton of variations in network topology and infra that it should (theoretically) work on. I imagine if you stay on the happy path everything works out just fine with Consul (even then, maybe only most of the time). The problem is that they don't spell out what the happy path is, and that all the other knobs they expose off to the side are actually down paths beleagured by dragons.

Re: Fly.io Status – Consul cluster outage

#55

Post by atonse »

Mizza wrote: post #52
atonse wrote: post #45

Earlier quoted context omitted.

We had mysterious consul outages (and related nomad outages) causing us to never deploy our new hashicorp stack to production. Shame cuz we were excited about our nomad+consul+vault setup and invested a lot of money into building it. But just didn’t have the time or enough depth of expertise to babysit it.

From my experience with the Hashi stack, I don't think it's a coincidence that Fly has a lot of downtime and are a major Hashi user. Terraform makes excellent bait though. Still love using Fly, please add static assets hosting/CDN.

How is this possible? How is consul not self—healing? It just seems so brittle in a way even database clusters aren’t.

Re: Fly.io Status – Consul cluster outage

#56

Post by lghh »

mrkurt wrote: post #48
candiddevmike wrote: post #39

Earlier quoted context omitted.

Are they really building everything in hard mode or do they just have a bad architecture?

Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".

I really appreciate the honesty here. I’m not a fly customer (no use case for it) but your transparency is admirable. Wish y’all the best.

Re: Fly.io Status – Consul cluster outage

#57

Post by mrcwinn »

mrkurt wrote: post #38
mrcwinn wrote: post #23

Earlier quoted context omitted.

Same. I’m so disappointed because I’ve been rooting for them. We were close to a major deployment/migration (well, major as is mid four figures per month, not major like Google) but they were removed from the decision set. It would not have been responsible to bet on them at this time. I hope they get this sorted - they’re really good folks!

Thank you! I'm both sorry it didn't work out (because $$$$) and also glad we didn't create any agony for you. Someday, we hope to create mild irritation for you, though, if we can.

Thanks for replying and saying that.

Re: Fly.io Status – Consul cluster outage

#58

Post by Benjamin_Dobell »

satvikpendem wrote: post #20

At this point I'm not sure why one wouldn't use something like Hetzner and slap Coolify or Dokku or something else on it.

You're right. We've been on Fly.io for 6 months[1] and it's been nothing but pain. ~10 years ago I took a start-up off an EC2 distributed set-up and moved them to a simple Dokku & Linode single VPS infra (plus separate staging env - https://github.com/glassechidna/dokku-graduate). Most content was served from S3 via a CDN, so workload was light. That simple VPS set up was super reliable and served us well for over 5 years. We eventually outgrew the infra and deployed a K8s cluster (on AWS). I left a short while after we were acquired, but I believe the K8s infra is holding strong. Unfortunately, this latest generation of PaaS really aren't living up to expectations.

[1] We're using so little infra at present that we're within their free usage tier. However, I want to clarify that this isn't because we aren't willing to pay, we specifically want to pay for reliable managed offerings. That's actually the entire point! If Fly.io can deliver on their vision, we'd gladly be billed at 100x the current usage rates.

Re: Fly.io Status – Consul cluster outage

#59

Post by 3np »

mrkurt wrote: post #48
candiddevmike wrote: post #39

Earlier quoted context omitted.

Are they really building everything in hard mode or do they just have a bad architecture?

Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".

As a fellow hashistack operator I'd love to hear what the bad decision was.

Re: Fly.io Status – Consul cluster outage

#60

Post by candiddevmike »

atonse wrote: post #55
Mizza wrote: post #52

Earlier quoted context omitted.

From my experience with the Hashi stack, I don't think it's a coincidence that Fly has a lot of downtime and are a major Hashi user. Terraform makes excellent bait though. Still love using Fly, please add static assets hosting/CDN.

How is this possible? How is consul not self—healing? It just seems so brittle in a way even database clusters aren’t.

Consul is a lot more than just a database cluster, and that may be part of its problem.
Post reply on HN