Volume Monitoring in Kubernetes with Prometheus
blog.argoproj.io
Volume Monitoring in Kubernetes with Prometheus
1–8 of 8 posts
Re: Volume Monitoring in Kubernetes with Prometheus
#2Now I just need a flexdriver which receives the warning and activates the code to create a new ebs volume and adds it to whatever lm k8s volume. EBS gp2 is uselessly slow without just combining them into one super danger mount. As with everything else in amazon, never use one of x. Whenever I use S3 and EBS i put an abstraction between me and them because amazon.
Re: Volume Monitoring in Kubernetes with Prometheus
#3Wouldn't it be better to run node_exporter (with only the fs metrics enabled) in a sidecar container in the pod that mounts the volume? That way, you can get the metrics scraped with the right pod labels. If you do it by node, you will only get the node name as a label.
Re: Volume Monitoring in Kubernetes with Prometheus
#4kubelet v1.8 collects volume stats, the proposal linked in the summary was forked, and the fork accepted & implemented: https://github.com/kubernetes/community/pull/930
Re: Volume Monitoring in Kubernetes with Prometheus
#5Wouldn't it be better to run node_exporter (with only the fs metrics enabled) in a sidecar container in the pod that mounts the volume? That way, you can get the metrics scraped with the right pod labels. If you do it by node, you will only get the node name as a label.
Yes. There is a disadvantage, that when your app already exposes metrics, you will have to tell prometheus' service discovery that you want to scrape two ports.
Re: Volume Monitoring in Kubernetes with Prometheus
#6Hi, node exporter maintainer here.
Yes, running the node exporter in a container is still tricky which is why we recommend running it straight on the host. I like the nsenter idea, wondering if it could be integrated without shelling out but I think there were some inherit problems with Goroutines and nsenter preventing that. If your interested in exploring this, feel free to open an github issues. Even better though would be support of unisolated workloads in kubernetes, similar to rkt stage1.
Re: Volume Monitoring in Kubernetes with Prometheus
#7>" In this post, we discussed monitoring volumes on Kubernetes using Prometheus, node-exporter, and nsenter. One disadvantage of this approach is that node-exporter must be run as a privileged container. In the future, it may be better to have Kubelet collect the volume stats so that we can get the stats directly from Kubernetes instead of running node-exporter as a privileged container."
I thought that node-exporter was meant to run in a container or on the host itself. Are there any potential downside to the single node-exporter in a side car approach?
Re: Volume Monitoring in Kubernetes with Prometheus
#8Wouldn't it be better to run node_exporter (with only the fs metrics enabled) in a sidecar container in the pod that mounts the volume? That way, you can get the metrics scraped with the right pod labels. If you do it by node, you will only get the node name as a label.
Yes. There is a disadvantage, that when your app already exposes metrics, you will have to tell prometheus' service discovery that you want to scrape two ports.
Well, you can easily set up Prometheus with a rule that automatically scrapes any discovered service (or pod) where the port name is =~ /^(exporter|.*-exporter)$/, for example. This allows you to name your port just "exporter", or you can have multiple ports named things like "volume-exporter".