We are rolling out Kubernetes to production next month and I'm interested to hear from people who made that step already.
Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
1–10 of 30 posts
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#2See the excellent collection of Kubernetes incident reviews at https://k8s.af
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#3Don't give everybody prod access, but give enough people prod access.
Use namespaces and logically bounded clusters. Get your monitoring, and tracing and a dashboard to visualize this figured out now.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#4- Deleting or bulk changing something? Always use the flag --record. This way, you can refer back to what you changed using kubectl rollout history.
- If you're planning on using GKE, you'll have to expose your apps using Ingress (this way you can use GCP's L7 Load Balancing with HTTPs). However, this architecture has many limits (e.g. a hard limit of 1000 forwarding rules (FW) per project, each ingress creates an FW and k8s ingress can't refer to another namespace), so make sure you use namespaces wisely.
- Try to learn and teach people on your team about requests and limits. If you don't use it carefully, you'll end up wasting a lot of resources. Also, make sure you have Prometheus and Grafana set up, to give you some visibility.
- Setup Heptio's Velero, it's a lifesaver, especially when running in a managed environment where you have no access to etcd. It can be used to backup your whole cluster and migrate workloads between clusters. If, for some reason, you end up deleting a cluster by mistake, it will be easier to recover its workloads using Velero.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#5DON'T: use it.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#6DON'T: use it.
You're gonna get downvoted to hell. But this is probably the sanest advice.
To OP- elaborate on what you're actually trying to accomplish
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#7We experienced an issue with a validating webhook controller configured to validate (way) more than needed - I wrote it up here: https://blog.jetstack.io/blog/gke-webhook-outage. It's on https://k8s.af - a great place for k8s related postmortems.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#8- Deleting or bulk changing something? Always use the flag --record. This way, you can refer back to what you changed using kubectl rollout history. - If you're planning on using GKE, you'll have to expose your apps using Ingress (this way you can use GCP's L7 Load Balancing with HTTPs). However, this architecture has many limits (e.g. a hard limit of 1000 forwarding rules (FW) per project, each ingress creates an FW…
Minor note: you can have multiple ingress resources for the same hostname. This way you can route some paths to some services in a namespace and other paths to other namespaces.
(Yes, it's confusing. Yes, it can be dangerous)
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#9- Deleting or bulk changing something? Always use the flag --record. This way, you can refer back to what you changed using kubectl rollout history. - If you're planning on using GKE, you'll have to expose your apps using Ingress (this way you can use GCP's L7 Load Balancing with HTTPs). However, this architecture has many limits (e.g. a hard limit of 1000 forwarding rules (FW) per project, each ingress creates an FW…
Minor note: you can have multiple ingress resources for the same hostname. This way you can route some paths to some services in a namespace and other paths to other namespaces. (Yes, it's confusing. Yes, it can be dangerous)
It's really confusing. The PaaS we provide to our devs creates a namespace per app with an ingress (when the app is a web app). We have hit this limit and we were thinking about doing what you mentioned.
After reading this gigantic issue (https://github.com/kubernetes/kubernetes/issues/17088), we gave up and just created another GCP project.
Re: Ask HN: Do's/don'ts of working with Kubernetes you learned through experience?
#10Do keep a gitops folder / repository to keep your cluster in sync with expectations, do not let adhoc edits become the norm.
Use tools like kustomize to reduce proliferation of duplicate k8s resource files.
Do make sure you are using health and liveness checks.
Definitely take care to specify resource requests and limits.
Do use annotations to control provider resources, rather than manually tweaking provider resources that are auto generated by basic k8s files with no annotations.
Aggregate your logs.