Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Status – Consul cluster outage

status.flyio.net

Post reply on HN
Print view View original

61–70 of 123 posts

Re: Fly.io Status – Consul cluster outage

#61

Post by fastest963 »

pcthrowaway wrote: post #53
HL33tibCe7 wrote: post #3

Earlier quoted context omitted.

The latter (they openly admit as such)

They say that, but they're also being actively supported by Hashicorp right now (one would presume), so they really need to maintain a good working relationship. I don't have a relationship with Hashicorp, and have tried using Consul. Everything about it is amazing in theory, but you might need a few years of experience with kube, consul, go, and maybe even the hashicorp stack to even begin debugging when things don'…

To add a data point we've been using Consul globally for several years now without any major outages. We do close to 50k qps with Consul at peak running on single digit cores per DC.

Re: Fly.io Status – Consul cluster outage

#62

Post by wjossey »

mrkurt wrote: post #32

This has been a rough week, and I'm sorry we broke peoples' apps. We had a big Nomad outage on Monday, and then a suspiciously similar Consul outage today. Both tipped over faster than we could detect and mitigate, and we ended up having to do serious surgery to build entirely new Consul/Nomad clusters. There's nothing to brag about here, I just wanted to let y'all know we're listening (even when things aren't on the…

This stuff is hard. As someone who runs infra teams for a living, these are the worst kinds of weeks.

Hang in there. You all will learn from this and be better for it. Your architecture will improve. Customers will give you a second chance. This too shall pass.

Sending positive vibes.

Re: Fly.io Status – Consul cluster outage

#63

Post by convolvatron »

pcthrowaway wrote: post #47
kbumsik wrote: post #36

I have seen some issues around Consul these days. As a person with no background in distributed systems, I am wondering why people choose Consul over alternatives. Are there features that etcd doesn't offer?

Etcd is really only for basic config. If you want apps to discover each other and be able to communicate effortlessly, even across datacenters, Consul, in theory, enables this. I say in theory because I couldn't get federated Consul actually working.

discovery isn't that hard a problem that you should cede your agency to a external party like Hashicorp

I used consul for a clustered service once, it was worth it for bringup. but I when I had problems I just wrote one in a couple days since I'd done so several times before. and it didn't fail for all the years that product was running.

Re: Fly.io Status – Consul cluster outage

#64

Post by alxmng »

satvikpendem wrote: post #20

At this point I'm not sure why one wouldn't use something like Hetzner and slap Coolify or Dokku or something else on it.

This. It’s way more performant too, because you can host DB and other services from the same machine.

You don’t need to orchestrate a complex cluster to serve thousands or even millions of users. You can scale to hundreds of gigs of memory on a single machine nowadays.

Re: Fly.io Status – Consul cluster outage

#65

Post by satvikpendem »

Benjamin_Dobell wrote: post #58
satvikpendem wrote: post #20

At this point I'm not sure why one wouldn't use something like Hetzner and slap Coolify or Dokku or something else on it.

You're right. We've been on Fly.io for 6 months[1] and it's been nothing but pain. ~10 years ago I took a start-up off an EC2 distributed set-up and moved them to a simple Dokku & Linode single VPS infra (plus separate staging env - https://github.com/glassechidna/dokku-graduate ). Most content was served from S3 via a CDN, so workload was light. That simple VPS set up was super reliable and served us well for over 5…

I'm waiting for Coolify's Kubernetes support, personally, I'd love to use it as a pseudo-managed service while still having much lower costs and higher uptime.

Re: Fly.io Status – Consul cluster outage

#66

Post by aeyes »

mrkurt wrote: post #46
kbumsik wrote: post #36

I have seen some issues around Consul these days. As a person with no background in distributed systems, I am wondering why people choose Consul over alternatives. Are there features that etcd doesn't offer?

We chose Nomad and adopted Consul as a result. Nomad and Consul work well together. I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.

Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems.

Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.

Re: Fly.io Status – Consul cluster outage

#67

Post by abofh »

akerl_ wrote: post #13
abofh wrote: post #8

Earlier quoted context omitted.

The former ish -- they relied on consul marketing that the hammer fit the square hole. Hashicorp has been pretty bad about marketing themselves as the right tool for any job, but they really only fit the narrowest of tasks before you find yourself needing an alternative or being compelled to buy a support contact. It's atlassian from Arkansas, just faster

Can you cite this? Because it seems like the opposite is true: https://news.ycombinator.com/item?id=35048318

You seem to have cited exactly why I don't recommend my clients use a hashi stack, it seems like you've failed to make a point?

Re: Fly.io Status – Consul cluster outage

#68

Post by aeyes »

mrkurt wrote: post #35
abofh wrote: post #7

Earlier quoted context omitted.

They just hired their first if I recall correctly. I feel for their customers more than I do for their shareholders

We've scaled infra ops from 3 to 7 people in the past few weeks. Our very first VP was a VP Infra Ops, because that's the thing we have to get best at to succeed as a business. Note that we grew the whole company from 25 to 60 over the last six months.

You might want to slow down on hiring, more people doesn't equal to solving the problem faster or better. It could be better to queue new sign ups to your service for a while, even if it's painful.

Re: Fly.io Status – Consul cluster outage

#69

Post by abofh »

mrkurt wrote: post #35
abofh wrote: post #7

Earlier quoted context omitted.

They just hired their first if I recall correctly. I feel for their customers more than I do for their shareholders

We've scaled infra ops from 3 to 7 people in the past few weeks. Our very first VP was a VP Infra Ops, because that's the thing we have to get best at to succeed as a business. Note that we grew the whole company from 25 to 60 over the last six months.

You've been around for a decade, I'm not giving you pity points. Can you provide what we want or not?

Re: Fly.io Status – Consul cluster outage

#70

Post by rtpg »

satvikpendem wrote: post #20

At this point I'm not sure why one wouldn't use something like Hetzner and slap Coolify or Dokku or something else on it.

Ops from scratch are annoying compared to the theoretical niceness of just pushing up a docker image.

Though I think a lot of this is incidental to just not really knowing the deal, and ops from scratch mean you have to make a lot of tiny decisions like "OK how do I get this package over here, how do I set it up, do I wipe the VM on OS-level udpates, do I need scripts for resetting the machine..." Having pre-made decisions for a bunch of questions means you aren't spending a bunch of time on tedious stuff when starting up a project.

Post reply on HN