Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Status – Consul cluster outage

status.flyio.net

Post reply on HN
Print view View original

81–90 of 123 posts

Re: Fly.io Status – Consul cluster outage

#81

Post by kbumsik »

aeyes wrote: post #66
mrkurt wrote: post #46

Earlier quoted context omitted.

We chose Nomad and adopted Consul as a result. Nomad and Consul work well together. I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.

Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems. Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.

Doesn't Consul have the similar storage limit btw?

I have seen very few strongly consistent distributed KV store that scales beyond 10GB+

Re: Fly.io Status – Consul cluster outage

#82

Post by mrkurt »

rtpg wrote: post #71
mrkurt wrote: post #35

Earlier quoted context omitted.

We've scaled infra ops from 3 to 7 people in the past few weeks. Our very first VP was a VP Infra Ops, because that's the thing we have to get best at to succeed as a business. Note that we grew the whole company from 25 to 60 over the last six months.

No offense but it's kind of wild to me that y'all had 3 infra operations people out of 60 hires. As someone who had to do SRE-style work in a smaller company for a long time despite obsetnsibly being a backend dev, the institutional knowledge you get from "real" SRE people is so valuable, and makes me a bit hopeful for the future.

It was really 7 out of 60, but yes. We ran into two problems: 1) the infra ops jobs is pretty intensely difficult here and 2) actually building an infra ops _org_ instead of just hiring individual engineers and overloading them seemed important.

Re: Fly.io Status – Consul cluster outage

#83

Post by abofh »

akerl_ wrote: post #76
abofh wrote: post #67

Earlier quoted context omitted.

You seem to have cited exactly why I don't recommend my clients use a hashi stack, it seems like you've failed to make a point?

> I have only positive things to say about every HashiCorp product I've worked with since I got here. You're making a claim that Hashicorp sold themselves as being a solution for problems that they can't solve. But the comment from an actual Fly.io employee suggests that isn't the case. They're stating that Fly.io pushed the product beyond its limits, and they don't seem to be projecting any of that as being the faul…

You literally started with a quote that I didn't write.

Hashi is dishonest and diminuative, providing products that generally should've been written off as a loss

Edit: We're apparently not allowed to interact beyond three replies: I have no beef with any hashi product that is satisfactory.

Your reply quotation explanation is unsatisfactory, you tried to quote something into a thread in the most irresponsible way you could - got called out for it and tried to top post to make it work.

I have never had a client using a hashi stack that was happy about it: price, quality or reliability it's a failure

I don't begrudge their work, their work is just subpar Quote that if you'd like, I won't interact with someone that starts with fraudulent misrepresentation.

Re: Fly.io Status – Consul cluster outage

#84

Post by abofh »

mrkurt wrote: post #74
abofh wrote: post #69

Earlier quoted context omitted.

You've been around for a decade, I'm not giving you pity points. Can you provide what we want or not?

Not right now, no. We've been running this product for about 2.5 years.

So who's product have I been using?

Re: Fly.io Status – Consul cluster outage

#85

Post by akerl_ »

abofh wrote: post #83
akerl_ wrote: post #76

Earlier quoted context omitted.

> I have only positive things to say about every HashiCorp product I've worked with since I got here. You're making a claim that Hashicorp sold themselves as being a solution for problems that they can't solve. But the comment from an actual Fly.io employee suggests that isn't the case. They're stating that Fly.io pushed the product beyond its limits, and they don't seem to be projecting any of that as being the faul…

You literally started with a quote that I didn't write. Hashi is dishonest and diminuative, providing products that generally should've been written off as a loss Edit: We're apparently not allowed to interact beyond three replies: I have no beef with any hashi product that is satisfactory. Your reply quotation explanation is unsatisfactory, you tried to quote something into a thread in the most irresponsible way you…

It’s a quote from the comment I linked, from the Fly.io employee, describing their opinion of Hashicorp products based on their usage of those products at Fly.io.

It seems pretty clear you have beef with Hashicorp and with their products. It’s entirely possible you’re right. But your original claim, that I was replying to, attempted to answer a question about why Fly.io was experiencing issues with Hashicorp products. And your answer doesn’t line up with clear public statements from Fly.io staff.

Re: Fly.io Status – Consul cluster outage

#86

Post by mdaniel »

atonse wrote: post #55
Mizza wrote: post #52

Earlier quoted context omitted.

From my experience with the Hashi stack, I don't think it's a coincidence that Fly has a lot of downtime and are a major Hashi user. Terraform makes excellent bait though. Still love using Fly, please add static assets hosting/CDN.

How is this possible? How is consul not self—healing? It just seems so brittle in a way even database clusters aren’t.

I haven't been responsible for babysitting consul, but I have been responsible for etcd for years and if consul's problem is anything like etcd's it's because members have identity and if one of them goes toes up then etcd will wait forever for the snowflake to come back to life, and if that's not how the underlying infra is configured, that's very very bad. Mix it mTLS into this story and it gets worse

I stayed away from the so-called "stacked" control plane of etcd inside kubernetes because it can make a tiny fire into a sharkfirenado but recently I've heard discussions of k3s (which uses dqlite) managing the etcd members and then "formal" kubernetes managing the workloads pointed at that k3s-stacked-etcd but I haven't tried it yet in order to know how theory and practice differ

Re: Fly.io Status – Consul cluster outage

#87

Post by mdaniel »

aeyes wrote: post #66
mrkurt wrote: post #46

Earlier quoted context omitted.

We chose Nomad and adopted Consul as a result. Nomad and Consul work well together. I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.

Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems. Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.

I would need a citation on the kubernetes control plane having any such hard limit. etcd is its own little snowflake, and I could very easily imagine it having some bad default value like that, or even kubeadm improperly configuring it

However, related to that, for big-time clusters (q.v. https://news.ycombinator.com/item?id=35174655 and https://news.ycombinator.com/item?id=25907312) one should without question move events over into their own etcd cluster: https://openai.com/research/scaling-kubernetes-to-2500-nodes...

Re: Fly.io Status – Consul cluster outage

#89

Post by ivzhh »

aeyes wrote: post #66
mrkurt wrote: post #46

Earlier quoted context omitted.

We chose Nomad and adopted Consul as a result. Nomad and Consul work well together. I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.

Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems. Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.

ByteDance replaces etcd with kubebrain [1], which is backed by their own KV store (TiKV seems also supported).

The single-group raft is the hard limit.

[1]: https://github.com/kubewharf/kubebrain

Re: Fly.io Status – Consul cluster outage

#90

Post by mrkurt »

phphphphp wrote: post #88
mrkurt wrote: post #74

Earlier quoted context omitted.

Not right now, no. We've been running this product for about 2.5 years.

https://fly.io/about/

Oh yes, I should edit. We launched this product a little over 2.5 years ago. We built a bunch of different things no one really wanted before that.
Post reply on HN