21–30 of 107 posts
What do you all do with the collected metrics over time? Do you store everything forever, drop everything after a couple weeks, or something on between? I've heard of people thinning out old data a bit (?) and storing it long term rather than storing everything. What's the usual thing people do?
Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
I found this interesting. My plan is to move from Prom to Victoria. https://medium.com/@valyala/measuring-vertical-scalability-f...
Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…
Biggest issue I've had was an app that was accidentally publishing several thousand metrics which caused the default scrape timeout of 15s to kick in.
(It was publishing Kafka lag per consumer group per topic, which was fine and dandy, until someone released an app that runs about 500 instances at peak, and scaled up and down frequently, and had incorporated the pod id into the consumer group names, which led to Kafka tracking many, many, many consumer groups. Given that the consumers were low value anyway, we now just exclude them from having their lag tracked.)
Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…
Prometheus is designed to be "functionally sharded". You shouldn't be running one "mega prometheus". Often it's something like 1 Prometheus per-team, depending on the amount of metrics each produces.
You can use federation at lower resolutions or a one of the distributed setups (Thanos/Cortex) if you want to avoid extra storage or lower resolution that federation entails.
> But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives
Not to distract, but I think you meant resolution, not cardinality. Cardinality is the metadata like labels/dimensions. Resolution is the granularity in the time.
I have a love/hate relationship with Prometheus. If I had no budget for metrics its likely the thing I would reach for, but I’m dying for someone to open source a ‘next level’ metrics system (something like Monarch or Circonus but free). But woe betide the team that has to run it as a service. Not that other metrics systems are better but Prometheus can be brutal in that space. As a ‘squad level’ tool its really good…
Coming from a monitoring system that supports push and pull with elegant auto-discovery, we're struggling to work out a sane architecture around (effectively pull-only) Prometheus.
Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
Earlier quoted context omitted.
Prometheus "scales" really well, but it does so via segmentation and federation, rather than increasing the size of an e.g. cluster. Some use cases don't fit to that model, so projects like Cortex and Thanos exist.
not vertically at least. the memory usage for indexing has room for improvement. If I read the pprofs correctly, every scrape interval and every remote write allocates huge amounts of memory which is only cleaned up on garbage collection. You can easily need >64 gb ram for tenthousands of time series, otherwise you oom.
We've got a somewhat similar landscape, on a pretty sizeable network - big investment in Zabbix and looking to move, perhaps slowly and perhaps only in part, towards Prometheus. Coming from a monitoring system that supports push and pull with elegant auto-discovery, we're struggling to work out a sane architecture around (effectively pull-only) Prometheus.