Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Status – Consul cluster outage

status.flyio.net

Post reply on HN
Print view View original

71–80 of 123 posts

Re: Fly.io Status – Consul cluster outage

#71

Post by rtpg »

mrkurt wrote: post #35
abofh wrote: post #7

Earlier quoted context omitted.

They just hired their first if I recall correctly. I feel for their customers more than I do for their shareholders

We've scaled infra ops from 3 to 7 people in the past few weeks. Our very first VP was a VP Infra Ops, because that's the thing we have to get best at to succeed as a business. Note that we grew the whole company from 25 to 60 over the last six months.

No offense but it's kind of wild to me that y'all had 3 infra operations people out of 60 hires.

As someone who had to do SRE-style work in a smaller company for a long time despite obsetnsibly being a backend dev, the institutional knowledge you get from "real" SRE people is so valuable, and makes me a bit hopeful for the future.

Re: Fly.io Status – Consul cluster outage

#72

Post by sergiomattei »

I’m rooting for Fly. I use them myself for a project, and love the service.

However, their transparency into outages and service rough edges is a double-edged sword: they’re building a reputation for unreliable software. It’s a shame to see this major outage happen right after last week’s post, it almost confirms the stereotype.

However, even with these flaws, I still think they’re building the best hosting out there. They’re taking bold risks and doing what others aren’t. I wish them the best.

Re: Fly.io Status – Consul cluster outage

#73

Post by throwawaaarrgh »

atonse wrote: post #55
Mizza wrote: post #52

Earlier quoted context omitted.

From my experience with the Hashi stack, I don't think it's a coincidence that Fly has a lot of downtime and are a major Hashi user. Terraform makes excellent bait though. Still love using Fly, please add static assets hosting/CDN.

How is this possible? How is consul not self—healing? It just seems so brittle in a way even database clusters aren’t.

All distributed decentralized systems are brittle. The only people who don't think this are people who haven't run them at scale.

Also, "self-healing" isn't really one thing. There are hundreds of different problems that can take out such a cluster, and every single one of them needs its own "self-healing" mechanism. These systems are literally the most complicated kinds of systems.

Re: Fly.io Status – Consul cluster outage

#74

Post by mrkurt »

abofh wrote: post #69
mrkurt wrote: post #35

Earlier quoted context omitted.

We've scaled infra ops from 3 to 7 people in the past few weeks. Our very first VP was a VP Infra Ops, because that's the thing we have to get best at to succeed as a business. Note that we grew the whole company from 25 to 60 over the last six months.

You've been around for a decade, I'm not giving you pity points. Can you provide what we want or not?

Not right now, no. We've been running this product for about 2.5 years.

Re: Fly.io Status – Consul cluster outage

#75

Post by satvikpendem »

rtpg wrote: post #70
satvikpendem wrote: post #20

At this point I'm not sure why one wouldn't use something like Hetzner and slap Coolify or Dokku or something else on it.

Ops from scratch are annoying compared to the theoretical niceness of just pushing up a docker image. Though I think a lot of this is incidental to just not really knowing the deal, and ops from scratch mean you have to make a lot of tiny decisions like "OK how do I get this package over here, how do I set it up, do I wipe the VM on OS-level udpates, do I need scripts for resetting the machine..." Having pre-made dec…

I can push up a Docker image or git push to deploy just fine with Coolify or Dokku. I've been using them for my projects for a while with no trouble, plus they're cheaper and more performant than paid PaaS.

Re: Fly.io Status – Consul cluster outage

#76

Post by akerl_ »

abofh wrote: post #67
akerl_ wrote: post #13

Earlier quoted context omitted.

Can you cite this? Because it seems like the opposite is true: https://news.ycombinator.com/item?id=35048318

You seem to have cited exactly why I don't recommend my clients use a hashi stack, it seems like you've failed to make a point?

> I have only positive things to say about every HashiCorp product I've worked with since I got here.

You're making a claim that Hashicorp sold themselves as being a solution for problems that they can't solve. But the comment from an actual Fly.io employee suggests that isn't the case. They're stating that Fly.io pushed the product beyond its limits, and they don't seem to be projecting any of that as being the fault of Hashicorp the company or of their products.

Re: Fly.io Status – Consul cluster outage

#77

Post by mrkurt »

wjossey wrote: post #62
mrkurt wrote: post #32

This has been a rough week, and I'm sorry we broke peoples' apps. We had a big Nomad outage on Monday, and then a suspiciously similar Consul outage today. Both tipped over faster than we could detect and mitigate, and we ended up having to do serious surgery to build entirely new Consul/Nomad clusters. There's nothing to brag about here, I just wanted to let y'all know we're listening (even when things aren't on the…

This stuff is hard. As someone who runs infra teams for a living, these are the worst kinds of weeks. Hang in there. You all will learn from this and be better for it. Your architecture will improve. Customers will give you a second chance. This too shall pass. Sending positive vibes.

<3

Re: Fly.io Status – Consul cluster outage

#79

Post by kbumsik »

mrkurt wrote: post #46
kbumsik wrote: post #36

I have seen some issues around Consul these days. As a person with no background in distributed systems, I am wondering why people choose Consul over alternatives. Are there features that etcd doesn't offer?

We chose Nomad and adopted Consul as a result. Nomad and Consul work well together. I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.

Thanks for the answer!

AFAIK doesn't Consul also use Raft?

Re: Fly.io Status – Consul cluster outage

#80

Post by mrkurt »

atonse wrote: post #50
mrkurt wrote: post #38

Earlier quoted context omitted.

Thank you! I'm both sorry it didn't work out (because $$$$) and also glad we didn't create any agony for you. Someday, we hope to create mild irritation for you, though, if we can.

I am feeling similarly. We’ve got a few apps in fly and have convinced devs to use it for their side projects. We’re excited about the promise of fly and were considering the HIPAA plan. But these stability issues actually make me more nervous about the fact that I’d have to manage my own postgres cluster and have to learn how to recover it in such an event. AWS RDS has made me soft! Wishing you guys the best. We’ll…

I appreciate that. I think we'll be in a really good spot in ~60 days.
Post reply on HN