Earlier quoted context omitted.
There was a blogpost about creating an alternative to etcd for super high scale kubernetes cluster. All code was open too. It was from someone named Benjamin I think but not sure. I’m not able to find the blogpost but maybe someone else can!
This might be what you're thinking of: https://bchess.github.io/k8s-1m/
Building the largest known Kubernetes cluster
91–94 of 94 posts
Re: Building the largest known Kubernetes cluster
#92Earlier quoted context omitted.
For those not aware, if you create too many resources you can easily use up all of the 8GB hard coded maximum size in etcd which causes a cluster failure. With compaction and maintenance this risk is mitigated somewhat but it just takes one misbehaving operator or integration (e.g. hundreds of thousands of dex session resources created for pingdom/crawlers) to mess everything up. Backups of etcd are critical. That de…
It's not hardcoded and you can increase it via flag.
Re: Building the largest known Kubernetes cluster
#93Earlier quoted context omitted.
It's not hardcoded and you can increase it via flag.
There is a hard coded warning which says safety not guaranteed after 8GB. I have tried increasing this after a database has become full and it didn’t start. It’s definitely not a recovery strategy for a full etcd by itself, maybe as part of a way to eek out a little larger margin of safety.
Re: Building the largest known Kubernetes cluster
#94Earlier quoted context omitted.
There is a hard coded warning which says safety not guaranteed after 8GB. I have tried increasing this after a database has become full and it didn’t start. It’s definitely not a recovery strategy for a full etcd by itself, maybe as part of a way to eek out a little larger margin of safety.
This warning seems to be outdated. We had run etcd at much larger volumes without issues (at least without issues related to its size). Alibaba has been running 100G etcd clusters for a while now, probably others too