Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Status – Consul cluster outage

status.flyio.net

Post reply on HN
Print view View original

91–100 of 123 posts

Re: Fly.io Status – Consul cluster outage

#91

Post by pcthrowaway »

3np wrote: post #59
mrkurt wrote: post #48

Earlier quoted context omitted.

Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".

As a fellow hashistack operator I'd love to hear what the bad decision was.

Hopefully they'll blog about it at some point. At this point I might pay just for their blog posts, even though I'm not using fly (yet?) because it's certainly a more cost-effective way to learn about big architectural mistakes than the way they're going about it, which involves actually paying the sunk costs.

Re: Fly.io Status – Consul cluster outage

#92

Post by throwdbaaway »

gzer0 wrote: post #22

Interestingly, Roblox went down for 73 hours due to a "unique" issue with Consul as well [1]. Great read on how the issue was approached, handled, and ultimately remediated. [1] https://blog.roblox.com/2022/01/roblox-return-to-service-10-...

https://github.com/hashicorp/consul/pull/12080 - this should be the Consul issue that brought down Roblox

Re: Fly.io Status – Consul cluster outage

#93

Post by mrkurt »

3np wrote: post #59
mrkurt wrote: post #48

Earlier quoted context omitted.

Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".

As a fellow hashistack operator I'd love to hear what the bad decision was.

We got ourselves into a bind because of Nomad. We outlined a bunch of it here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchest...

The short version is that "using Nomad and Consul for the type of global workloads we run is not a good choice". I do not believe we'd have the same problems with Nomad + Consul in a single region. But running a single, global cluster of each of these is suboptimal.

The second problem was using some Consul features that forced us to keep it single region. What we actually need is a global view of a single service. Federated Consul doesn't quite give us that. Earlier versions of our infrastructure were using a bunch of Consul watches to update local state, so we couldn't really federate.

Some of this I'd do very differently if we rewound. But we were also building an idea with no actual users. Nomad and Consul gave us a nice platform to experiment on. We just outgrew the "prototype" as we learned what people actually wanted from us.

Re: Fly.io Status – Consul cluster outage

#94

Post by grrdotcloud »

atonse wrote: post #55
Mizza wrote: post #52

Earlier quoted context omitted.

From my experience with the Hashi stack, I don't think it's a coincidence that Fly has a lot of downtime and are a major Hashi user. Terraform makes excellent bait though. Still love using Fly, please add static assets hosting/CDN.

How is this possible? How is consul not self—healing? It just seems so brittle in a way even database clusters aren’t.

I've worked with Vault and clusters, not Consul specifically but generally there's a healthy cluster state until something happens putting the cluster into an inconsistent state.

Generally there's a master node or multiple nodes in agreement. If the cluster cannot agree on it's current state the entire system may run multiple versions or be completely unavailable or provide inconsistent response bringing down other systems that rely on it.

Inspection itself is hampered by elections or syncing state or other process/race related/caching/ddossing itself or other services.

Re: Fly.io Status – Consul cluster outage

#95

Post by cheeseprocedure »

mdaniel wrote: post #86
atonse wrote: post #55

Earlier quoted context omitted.

How is this possible? How is consul not self—healing? It just seems so brittle in a way even database clusters aren’t.

I haven't been responsible for babysitting consul, but I have been responsible for etcd for years and if consul's problem is anything like etcd's it's because members have identity and if one of them goes toes up then etcd will wait forever for the snowflake to come back to life, and if that's not how the underlying infra is configured, that's very very bad. Mix it mTLS into this story and it gets worse I stayed away…

Consul’s autopilot feature makes life a little easier by automatically reaping failed instances:

https://developer.hashicorp.com/consul/tutorials/datacenter-...

Paired with cloud discovery, it makes for a tolerable operational experience when instances are expected to occasionally disappear.

Re: Fly.io Status – Consul cluster outage

#96

Post by grrdotcloud »

mrkurt wrote: post #46
kbumsik wrote: post #36

I have seen some issues around Consul these days. As a person with no background in distributed systems, I am wondering why people choose Consul over alternatives. Are there features that etcd doesn't offer?

We chose Nomad and adopted Consul as a result. Nomad and Consul work well together. I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.

Raft is amazing and totally frustrating.

I think I understand how you're using it and curious if you've considered how AWS STS API manages their cross region syncing gets solved.

Re: Fly.io Status – Consul cluster outage

#97

Post by abofh »

mrkurt wrote: post #90
phphphphp wrote: post #88

Earlier quoted context omitted.

https://fly.io/about/

Oh yes, I should edit. We launched this product a little over 2.5 years ago. We built a bunch of different things no one really wanted before that.

You should edit, but that's not my puppet to control, I only use this one.

So yeah, you've built a ton of shit nobody wanted as a product, been there, done that. You've convinced me fly doesn't fit business, we're done.

Re: Fly.io Status – Consul cluster outage

#98

Post by sidlls »

fastest963 wrote: post #61
pcthrowaway wrote: post #53

Earlier quoted context omitted.

They say that, but they're also being actively supported by Hashicorp right now (one would presume), so they really need to maintain a good working relationship. I don't have a relationship with Hashicorp, and have tried using Consul. Everything about it is amazing in theory, but you might need a few years of experience with kube, consul, go, and maybe even the hashicorp stack to even begin debugging when things don'…

To add a data point we've been using Consul globally for several years now without any major outages. We do close to 50k qps with Consul at peak running on single digit cores per DC.

50k qps on consul or system wide?

Re: Fly.io Status – Consul cluster outage

#99

Post by faizshah »

candiddevmike wrote: post #39
suryao wrote: post #18

Fly is building everything in hard mode - since they are not layering on top of an existing cloud like pretty much everyone else (heroku, render, railway, ...). It's either very smart (if they pull it off) because they will have a ginormous cost advantage or they fail. I'm personally of the opinion that the ux on top of aws/gcp/... is worse than a doo-doo in a shoe. However, they are as stable as can be (all complex…

Are they really building everything in hard mode or do they just have a bad architecture?

AWS invested a ton into limiting the blast radius of failures by isolating AZs, regions and using a cellular (service level isolated shards) architecture. I am surprised these ideas have not propagated to newer companies trying to build clouds: https://m.youtube.com/watch?v=swQbA4zub20

AWS isn’t perfect but these lessons were learned by fire because these sorts of global outages can seriously harm reputations.

Re: Fly.io Status – Consul cluster outage

#100

Post by throwdbaaway »

mrkurt wrote: post #42
pm90 wrote: post #33

> We are working to build a new Consul cluster with 10x the RAM. Oh boy. I wouldn’t wanna be the people doing this. Working with infrastructure is hard . Doing it under tight SLAs? Ugh. I really hope the people working on this are being well supported.

You wouldn't necessarily know this from the outside, but we have _exceptional_ internal support when things go sideways. This is relatively new, up until about two months ago most incidents were run by 1.5 people. We had 7 people working this one today.

I don't really know him, but from what I can tell, https://github.com/wjordan is at least equivalent to 2.0 people.
Post reply on HN