This shit is hard. Running a cloud service at one of the Big 3 is hard, I can’t imagine doing it with such a small team with your own infra.
Fly.io Status – Consul cluster outage
101–110 of 123 posts
Re: Fly.io Status – Consul cluster outage
#102Earlier quoted context omitted.
As a fellow hashistack operator I'd love to hear what the bad decision was.
We got ourselves into a bind because of Nomad. We outlined a bunch of it here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchest... The short version is that "using Nomad and Consul for the type of global workloads we run is not a good choice". I do not believe we'd have the same problems with Nomad + Consul in a single region. But running a single, global cluster of each of these is suboptimal. The second…
One of the single best design decisions that AWS makes is to isolate each region as much as possible so that truly global outages are nearly impossible.
Re: Fly.io Status – Consul cluster outage
#103Earlier quoted context omitted.
Are they really building everything in hard mode or do they just have a bad architecture?
AWS invested a ton into limiting the blast radius of failures by isolating AZs, regions and using a cellular (service level isolated shards) architecture. I am surprised these ideas have not propagated to newer companies trying to build clouds: https://m.youtube.com/watch?v=swQbA4zub20 AWS isn’t perfect but these lessons were learned by fire because these sorts of global outages can seriously harm reputations.
I feel that this is one of the biggest advantages AWS has over Azure. AWS has never had truly global outages the way that Azure has had with Azure AD
Re: Fly.io Status – Consul cluster outage
#104Been a fan of fly and have had most, if not all, of my side and semi-side projects on there for some time now. But...the ratio of good/fun/snarky blog posts to reliable service has gotten a bit too large for me, starting to look for other providers at this point just in case they can't turn this trend around. Honestly been a good object lesson for me in the importance of backing up marketing/hype/"mind-share" stuff w…
Wow, part of Delaware’s tax website was hanging on unpkg today, now I know why!
Re: Fly.io Status – Consul cluster outage
#105Earlier quoted context omitted.
AWS invested a ton into limiting the blast radius of failures by isolating AZs, regions and using a cellular (service level isolated shards) architecture. I am surprised these ideas have not propagated to newer companies trying to build clouds: https://m.youtube.com/watch?v=swQbA4zub20 AWS isn’t perfect but these lessons were learned by fire because these sorts of global outages can seriously harm reputations.
I feel that this is one of the biggest advantages AWS has over Azure. AWS has never had truly global outages the way that Azure has had with Azure AD
Azures global outages have all been DNS more or less.
And AWS has had a few of those in my 13-14 years I have used them :D
Azure reliability sucks more for sure though. Especiallly networking.
Edit: us-east-1 going down disrupts most of global AWS pretty severely fwiw.
Re: Fly.io Status – Consul cluster outage
#106Earlier quoted context omitted.
I feel that this is one of the biggest advantages AWS has over Azure. AWS has never had truly global outages the way that Azure has had with Azure AD
Azures global outages have all been DNS more or less. And AWS has had a few of those in my 13-14 years I have used them :D Azure reliability sucks more for sure though. Especiallly networking. Edit: us-east-1 going down disrupts most of global AWS pretty severely fwiw.
That actually brings up another aspect of AWS architecture, even for global services that depend on us-east-1, AWS separates the control plane (configuration of resources) from the data plane (usage of resources). These global services like IAM have regionalized data planes: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...
In a disaster scenario, the data plane operations can continue so customer workloads can still run while the control plane might experience downtime. This is another lesson where in the case of fly isolating the control plane (deployment of services) from the data plane (executing customer code) could have limited the blast radius of this fault instead of using a global cluster manager.
Re: Fly.io Status – Consul cluster outage
#107Earlier quoted context omitted.
Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems. Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.
I would need a citation on the kubernetes control plane having any such hard limit. etcd is its own little snowflake, and I could very easily imagine it having some bad default value like that, or even kubeadm improperly configuring it However, related to that, for big-time clusters (q.v. https://news.ycombinator.com/item?id=35174655 and https://news.ycombinator.com/item?id=25907312 ) one should without question move…
It’s etcd default https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi...
There’s also max object size of 1MB on the apiserserver side I believe
Re: Fly.io Status – Consul cluster outage
#108At this point I'm not sure why one wouldn't use something like Hetzner and slap Coolify or Dokku or something else on it.
Even with coolify or dokku you still have to manage the machine, those tools just simplify deployment. You still have to take care of host-level security and maintenance. Which is a hassle when all you really want is to stick your app somewhere and have it run.
Re: Fly.io Status – Consul cluster outage
#109Earlier quoted context omitted.
We chose Nomad and adopted Consul as a result. Nomad and Consul work well together. I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.
Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems. Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.
I’ve run it with over 50G under heavy load (10k+ qps) and it was fine. It’s pretty sensitive to disk latency though
Re: Fly.io Status – Consul cluster outage
#110Earlier quoted context omitted.
Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems. Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.
Doesn't Consul have the similar storage limit btw? I have seen very few strongly consistent distributed KV store that scales beyond 10GB+
It’s the underlying db limit (boltdb) which both etcd and consul use