11–20 of 107 posts
Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
(Disclaimer: I work at Timescale)
But woe betide the team that has to run it as a service. Not that other metrics systems are better but Prometheus can be brutal in that space.
As a ‘squad level’ tool its really good. After that it gets hairy fast.
I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely.
But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s).
Not to mention the lost writes (missing data points) which have no logged explanation.
All of this, however, was in my homelab, which, while unconstrained in resources lacks a lot of the fit and finish of a prod system.
I also take pause with the architecture; it’s not meant to scale. It’s written on the tin so it’s not like I’m picking fault, but when you’re building a dashboard that sucks in data from 25 different Prometheus data sources, it becomes difficult to run functions like SUM(), because the keys may be out of sync causing some really ugly and inaccurate representations of data.
Everything about the design (polling, single database) tells me that it was designed primarily to sit alongside something small. It could never handle the tens of millions of data points per second that I ingest(ed) at my (now previous) job.
But it has a lot of hype, and maybe I’m holding it wrong.
Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
https://medium.com/@valyala/measuring-vertical-scalability-f...
Earlier quoted context omitted.
Prometheus "scales" really well, but it does so via segmentation and federation, rather than increasing the size of an e.g. cluster. Some use cases don't fit to that model, so projects like Cortex and Thanos exist.
not vertically at least. the memory usage for indexing has room for improvement. If I read the pprofs correctly, every scrape interval and every remote write allocates huge amounts of memory which is only cleaned up on garbage collection. You can easily need >64 gb ram for tenthousands of time series, otherwise you oom.
Earlier quoted context omitted.
Prometheus "scales" really well, but it does so via segmentation and federation, rather than increasing the size of an e.g. cluster. Some use cases don't fit to that model, so projects like Cortex and Thanos exist.
not vertically at least. the memory usage for indexing has room for improvement. If I read the pprofs correctly, every scrape interval and every remote write allocates huge amounts of memory which is only cleaned up on garbage collection. You can easily need >64 gb ram for tenthousands of time series, otherwise you oom.
What do you all do with the collected metrics over time? Do you store everything forever, drop everything after a couple weeks, or something on between? I've heard of people thinning out old data a bit (?) and storing it long term rather than storing everything. What's the usual thing people do?
Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
I found this interesting. My plan is to move from Prom to Victoria. https://medium.com/@valyala/measuring-vertical-scalability-f...