The Guide to Kubernetes Labels • Hacker News Board

Re: The Guide to Kubernetes Labels

#11

Post by webb » Wed, Sep 01, 2021, 8:08 PM UTC

alexjplant wrote: post #6

Uhhh... > Instead, Kubernetes annotations are meant for sharing information with other colleagues. For example, you may use an annotation to record the phone number of the person responsible for a pod ...wat? The k8s docs say > You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. I've only ever seen them used in thi…

Yeah, I agree. I actually think that labels are a much better fit for this reference.

> Instead, Kubernetes annotations are meant for sharing information with other colleagues. For example, you may use an annotation to record the phone number of the person responsible for a pod

But it can be super helpful to put owner/contact info for alerting purposes as teams scale.

I also agree that having this metadata on the controller is helpful.

Re: The Guide to Kubernetes Labels

#12

Post by mikesabbagh » Wed, Sep 01, 2021, 8:11 PM UTC

some monitoring tools like kiali expect some standard labels to be available like app=nginx. This article misses out on the very standard label "app"

Re: The Guide to Kubernetes Labels

#13

Post by wdb » Wed, Sep 01, 2021, 8:21 PM UTC

GauntletWizard wrote: post #8

alexjplant wrote: post #6

Uhhh... > Instead, Kubernetes annotations are meant for sharing information with other colleagues. For example, you may use an annotation to record the phone number of the person responsible for a pod ...wat? The k8s docs say > You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. I've only ever seen them used in thi…

I wouldn't use it on a pod directly, but I recommend that annotations on deployments include the oncall rotation that is responsible for them - And there's no real reason not to cascade that down onto pods as well.

Interesting, do you have an example how this would look like?

Re: The Guide to Kubernetes Labels

#14

Post by brancz » Wed, Sep 01, 2021, 8:22 PM UTC

jrockway wrote: post #10

brancz wrote: post #7

Earlier quoted context omitted.

Unless you explicitly allow them (via a flag) in your kube-state-metrics deployment, Kubernetes labels and annotations are not translated into Prometheus labels.

kube-state-metrics doesn't specifically matter. Prometheus has built-in Kubernetes service discovery, and you can drop labels/annotations or not. Most people never see this because they take an opaque blob from a vendor (helm chart, prometheus-operator), etc. -- but it's there: https://prometheus.io/docs/prometheus/latest/configuration/c... kube-state-metrics does its own service discovery and outputs prometheus metr…

You’re absolutely right, though none of these transfer labels by default (neither does the example Prometheus config in the prometheus repo). Prometheus, including Prometheus operator require you to allow-list them. (I should have mentioned that I maintain the Prometheus Kubernetes SD, and am the original creator of kube-state-metrics, prometheus-operator and kube-prometheus)

Re: The Guide to Kubernetes Labels

#15

Post by dan_quixote » Wed, Sep 01, 2021, 8:25 PM UTC

alexjplant wrote: post #6

Uhhh... > Instead, Kubernetes annotations are meant for sharing information with other colleagues. For example, you may use an annotation to record the phone number of the person responsible for a pod ...wat? The k8s docs say > You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. I've only ever seen them used in thi…

Yup, used quite a lot for things like loadbalancer configuration (for instance): https://kubernetes-sigs.github.io/aws-load-balancer-controll...

Re: The Guide to Kubernetes Labels

#16

Post by alpb » Wed, Sep 01, 2021, 8:28 PM UTC

I've been working on Kubernetes stuff for about 5 years now and I have never seen people "organize" their stuff with labels.

It sounds great in theory ("lemme query all my pods in staging", "lemme find secrets by the marketing team") but it's just not used for some reason. I think people like flat organization (e.g. "find deployment X") more than searching by labels.

Re: The Guide to Kubernetes Labels

#17

Post by stumarkin » Wed, Sep 01, 2021, 8:47 PM UTC

alpb wrote: post #16

I've been working on Kubernetes stuff for about 5 years now and I have never seen people "organize" their stuff with labels. It sounds great in theory ("lemme query all my pods in staging", "lemme find secrets by the marketing team") but it's just not used for some reason. I think people like flat organization (e.g. "find deployment X") more than searching by labels.

Interesting! Does that present challenges when it comes to monitoring or managing at scale? Or when a project spans multiple clusters?

Re: The Guide to Kubernetes Labels

#18

Post by webb » Wed, Sep 01, 2021, 8:52 PM UTC

alpb wrote: post #16

I've been working on Kubernetes stuff for about 5 years now and I have never seen people "organize" their stuff with labels. It sounds great in theory ("lemme query all my pods in staging", "lemme find secrets by the marketing team") but it's just not used for some reason. I think people like flat organization (e.g. "find deployment X") more than searching by labels.

I've actually worked with 100+ companies that use labels heavily. It's way more common when you have multiple engineering teams and nobody everybody knows all projects in a cluster. Also, it's common to start with namespace labels which are generally easier to manage.

Re: The Guide to Kubernetes Labels

#19

Post by GauntletWizard » Wed, Sep 01, 2021, 9:00 PM UTC

wdb wrote: post #13

GauntletWizard wrote: post #8

Earlier quoted context omitted.

I wouldn't use it on a pod directly, but I recommend that annotations on deployments include the oncall rotation that is responsible for them - And there's no real reason not to cascade that down onto pods as well.

Interesting, do you have an example how this would look like?

Just "company.com/pagerduty-service" as a "required" annotation (i.e. if you forget your namespace's team gets a reminder at their E-mail, which is another required annotation)

Namespace onboarding is self-service by pull request: a template namespace is in the readme, and contains a number of standard annotations like ldap group and etc. The kubernetes admin team accepts and merges those requests and the bot deploys the namespace folder to the clusters.

This is basically zero overhead at small groups and scales well to giant companies, you just add annotations and sometimes labels as business requirements and structure dictate.

Re: The Guide to Kubernetes Labels

#20

Post by raitom » Wed, Sep 01, 2021, 9:38 PM UTC

Funny. I'm working on a specific issue with GCP GKE Autopilot. I'm running a bunch of workloads every day with 1 pod per client. Each pod has various running time and I wanted to know the pod cost per client.

I tried to use Kubecost but could not find any option at all to do what I wanted. My labels never appeared on the Kubecost UI. It's funny now to see an article about labels knowing that Kubecost itself doesn't seem to support them.