DON'T: use it.
Second that. Unless you really know what you are doing, don't use it.
Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
11–20 of 30 posts
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#12Don't use Kubernetes.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#13Can I be honest?
As someone who hopped on the K8s bandwagon back in the early days (circa-early 2017), _do not_ go into production with Kubernetes if you're still asking this question.
Just a few of the issues I've run into over the past 2 1/2 years or so:
- Kubernetes DNS flaking out completely
- Kubernetes DNS flaking out occasionally (for ~5 percent of queries)
- Giving out too many permissions, causing pods to be deleted without a clear reason why, often taking down production traffic or logging with it
- Giving out too few permissions, making our deployment infrastructure depend on a few lynchpins rather than sharing the production burden
- probably a dozen different logging aggregation systems, none of which strike a balance between speed and CPU cost
- probably a half-dozen different service meshes, all of which suck (with the exception of linkerd, which is actually quite good)
- teams with bad santization practices leaking credentials all over the place
- Running Vault in Kubernetes (really, don't ever do this)
- Disks becoming unattached from their pods for no discernable reason, only to be re-attached minutes later again with no explanation
- At least one major production outage on every single Kubernetes-based system I've built that can be directly attributed to Kubernetes
- Etcd failovers
- Etcd replication failures
- Privilege escalation due to an unsecured Jenkins builder causing credential exfiltration (this one was _super_ fun to fix)
Kubernetes is a powerful tool, and I've helped run some massive (1000+ node, 5000+ pod x 3 AZ's) systems based on K8s, but it took me a solid year of experimenting and tinkering to feel even remotely comfortable putting anything based on K8s into production. If you haven't run into any "major" issues, you're going to very soon. I can only wish you good luck.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#14Managed or custom deploy? What is the size of the cluster and team that will be using it?
Kubernetes is a hell of a lot configurable, so your environment matters a lot on the must and nice to have.
If not managed, make sure you go through all components flags and configure things like reserved resources, forbid hostpath usage, pod security policies (do not allow root), etc
Also, avoid service meshes until you fully understand how to use “vanilla” Kubernetes, don’t add this complexity from day 1 because debugging cluster issues can get a lot harder.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#15Do keep a gitops folder / repository to keep your cluster in sync with expectations, do not let adhoc edits become the norm. Use tools like kustomize to reduce proliferation of duplicate k8s resource files. Do make sure you are using health and liveness checks. Definitely take care to specify resource requests and limits. Do use annotations to control provider resources, rather than manually tweaking provider resourc…
Could you comment more on your log aggregation tooling? Did you set up an ELK stack? Self installed or paid? That is the most frustrating part for me — why some of the tooling is not built in.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#16We're moving away from Kubernetes, to Aptible.
If you're asking these kinds of questions you shouldn't be using kubernetes.
If you are going to use it, be ready to have an engineer on your time be full time devops. Or be ready to hire someone who knows k8. It'll be around 110k to 140k.
But really, don't use it. The gospel you hear is from engineers who already invested their careers in it. Buyer beware.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#17If your application requires high availability, make sure you are setting pod disruption budgets and have some special behavior when SIGKILL is sent to an app/pod. For some of our applications, we have some logic to finish all current requests after SIGKILL is sent, so that none are dropped.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#18If your application requires high availability, make sure you are setting pod disruption budgets and have some special behavior when SIGKILL is sent to an app/pod. For some of our applications, we have some logic to finish all current requests after SIGKILL is sent, so that none are dropped.
[deleted]
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#19If your application requires high availability, make sure you are setting pod disruption budgets and have some special behavior when SIGKILL is sent to an app/pod. For some of our applications, we have some logic to finish all current requests after SIGKILL is sent, so that none are dropped.
You mean SIGTERM, right? That's what gets sent to the containers before the grace period expires.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#20Do keep a gitops folder / repository to keep your cluster in sync with expectations, do not let adhoc edits become the norm. Use tools like kustomize to reduce proliferation of duplicate k8s resource files. Do make sure you are using health and liveness checks. Definitely take care to specify resource requests and limits. Do use annotations to control provider resources, rather than manually tweaking provider resourc…
Could you comment more on your log aggregation tooling? Did you set up an ELK stack? Self installed or paid? That is the most frustrating part for me — why some of the tooling is not built in.
We don't have the most sophisticated setup, we're on AWS so we use a fluentd daemonset to ship logs to cloudWatch.
In many respects kubernetes is just a really nice piece of marble to start with, you still have to carve your statue.
If certain tooling was built-in they would be making stack-specific subjective decisions for you, which is somewhat antithetical to the kubernetes model.
They'll make critical operational decisions for you (e.g. core competencies like scheduling work across a pool of resources), but when it comes to supporting tooling, you still have to make some decisions on your own.
For a lesser burden, there are most likely helm charts or other prepackaged log aggregation tools out there.