Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Status – Consul cluster outage

status.flyio.net

Post reply on HN
Print view View original

111–120 of 123 posts

Re: Fly.io Status – Consul cluster outage

#111

Post by satvikpendem »

overbytecode wrote: post #108
satvikpendem wrote: post #20

At this point I'm not sure why one wouldn't use something like Hetzner and slap Coolify or Dokku or something else on it.

Even with coolify or dokku you still have to manage the machine, those tools just simplify deployment. You still have to take care of host-level security and maintenance. Which is a hassle when all you really want is to stick your app somewhere and have it run.

Yeah I mean the host level maintenance isn't really a big issue, I can already stick my app somewhere and have it run, after the initial host setup. Maintenance afterwards is also pretty minimal.

Re: Fly.io Status – Consul cluster outage

#112

Post by mcsniff »

sergiomattei wrote: post #72

I’m rooting for Fly. I use them myself for a project, and love the service. However, their transparency into outages and service rough edges is a double-edged sword: they’re building a reputation for unreliable software. It’s a shame to see this major outage happen right after last week’s post, it almost confirms the stereotype. However, even with these flaws, I still think they’re building the best hosting out there…

> they’re building a reputation for unreliable software

This is a terrific way to word what might be happening unconsciously.

Fly posts about how hard things are during and after service outages -- while I also love the transparency, most people don't want to 'be a passenger on a plane that's being built while it's flying' especially when it comes to their business, myself included.

Re: Fly.io Status – Consul cluster outage

#113

Post by jen20 »

Mizza wrote: post #52
atonse wrote: post #45

Earlier quoted context omitted.

We had mysterious consul outages (and related nomad outages) causing us to never deploy our new hashicorp stack to production. Shame cuz we were excited about our nomad+consul+vault setup and invested a lot of money into building it. But just didn’t have the time or enough depth of expertise to babysit it.

From my experience with the Hashi stack, I don't think it's a coincidence that Fly has a lot of downtime and are a major Hashi user. Terraform makes excellent bait though. Still love using Fly, please add static assets hosting/CDN.

The simpler explanation is that running products designed for LAN usage on a WAN is a fundamentally bad plan, as the folks over at fly acknowledge, even in this thread.

Meanwhile, hundreds of thousands of Consul, Nomad and Vault clusters used appropriately work perfectly well…

Re: Fly.io Status – Consul cluster outage

#114

Post by jen20 »

luhn wrote: post #6

Relevant: "Reliability: It's not great" from last week https://news.ycombinator.com/item?id=35044516 They even specifically call out Consul as a source of trouble. > We propagate app instance and health information across all our regions. That’s how our proxies know where to route requests, and how our DNS servers know what names to give out. > We started out using HashiCorp Consul for this. But we were shoehorning C…

They call out THEIR USAGE of Consul as a source of trouble. This is quite different.

Re: Fly.io Status – Consul cluster outage

#115

Post by jen20 »

sidlls wrote: post #11
Thaxll wrote: post #2

They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?

Both, though the latter is likely due to marketing/promises from HashiCorp. Consul (and the entire hashicorp stack, really) is overengineered, under-optimized, and generally terrible to use at any scale beyond "small".

And yet the comment from a member of the actual team in question underneath says the opposite…

Re: Fly.io Status – Consul cluster outage

#116

Post by jen20 »

pcthrowaway wrote: post #53
HL33tibCe7 wrote: post #3

Earlier quoted context omitted.

The latter (they openly admit as such)

They say that, but they're also being actively supported by Hashicorp right now (one would presume), so they really need to maintain a good working relationship. I don't have a relationship with Hashicorp, and have tried using Consul. Everything about it is amazing in theory, but you might need a few years of experience with kube, consul, go, and maybe even the hashicorp stack to even begin debugging when things don'…

Out of interest why would one presume that they are being actively supported? I haven’t read everything about this saga, but I’ve never seen any mention of a commercial relationship.

Re: Fly.io Status – Consul cluster outage

#117

Post by srhyne »

wjossey wrote: post #62
mrkurt wrote: post #32

This has been a rough week, and I'm sorry we broke peoples' apps. We had a big Nomad outage on Monday, and then a suspiciously similar Consul outage today. Both tipped over faster than we could detect and mitigate, and we ended up having to do serious surgery to build entirely new Consul/Nomad clusters. There's nothing to brag about here, I just wanted to let y'all know we're listening (even when things aren't on the…

This stuff is hard. As someone who runs infra teams for a living, these are the worst kinds of weeks. Hang in there. You all will learn from this and be better for it. Your architecture will improve. Customers will give you a second chance. This too shall pass. Sending positive vibes.

Lovely response. Ah, kindness. So refreshing to see.

Re: Fly.io Status – Consul cluster outage

#118

Post by mrkurt »

throwdbaaway wrote: post #100
mrkurt wrote: post #42

Earlier quoted context omitted.

You wouldn't necessarily know this from the outside, but we have _exceptional_ internal support when things go sideways. This is relatively new, up until about two months ago most incidents were run by 1.5 people. We had 7 people working this one today.

I don't really know him, but from what I can tell, https://github.com/wjordan is at least equivalent to 2.0 people.

Accurate.

Re: Fly.io Status – Consul cluster outage

#119

Post by mdaniel »

ivzhh wrote: post #89
aeyes wrote: post #66

Earlier quoted context omitted.

Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems. Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.

ByteDance replaces etcd with kubebrain [1], which is backed by their own KV store (TiKV seems also supported). The single-group raft is the hard limit. [1]: https://github.com/kubewharf/kubebrain

That's interesting, thanks for the link. I held out high hopes for pluggable KV in kubernetes for the longest time, but since that issue was closed WONTFIX I resigned my hopes

Heh, that kubebrain TODO is some "oh, really?"

* Guarantee consistence in critical cases

but I give them huge props for calling out Jepsen

Re: Fly.io Status – Consul cluster outage

#120

Post by pcthrowaway »

jen20 wrote: post #116
pcthrowaway wrote: post #53

Earlier quoted context omitted.

They say that, but they're also being actively supported by Hashicorp right now (one would presume), so they really need to maintain a good working relationship. I don't have a relationship with Hashicorp, and have tried using Consul. Everything about it is amazing in theory, but you might need a few years of experience with kube, consul, go, and maybe even the hashicorp stack to even begin debugging when things don'…

Out of interest why would one presume that they are being actively supported? I haven’t read everything about this saga, but I’ve never seen any mention of a commercial relationship.

If you're on a slowly-sinking ship, it'd be silly not to at least try to bring in someone with the full context of the ship's architecture to get it serviceable before going all-in on the decision to engineer and build an entirely new kind of ship while still aboard the sinking one.
Post reply on HN