Live data from Hacker News

Big Prometheus: Thanos, Cortex, M3DB and VictoriaMetrics at Scale

monitoring2.substack.com

21–22 of 22 posts

Re: Big Prometheus: Thanos, Cortex, M3DB and VictoriaMetrics at Scale

#21
So here is my case. I'm running multiple Prometheus HA pairs to cover different teams. At the moment, I'm using Thanos and VictoriaMetrics in parallel to test them out.

Thanos was the first I set up as VM wasn't open-sourced yet. It wasn't hard to setup and had it running in about a day together with Minio as an S3 backend. To this day it's running without a problem, apart from an alarm every now and then that the Store or Compactor couldn't get something done. But I didn't look too much into it since everything graphing-wise seems to work. Upgrades are also easy and I love the global querier option. I sometimes see people having OOMs on a rather "large" servers on Slack, but Thanos team is suppose to be working on optimizing memory usage and it's getting better and better.

After the last PromCon, I also configured VictoriaMetrics. Installation was as simple as it can be, way simpler than Thanos, but I'm using a single node version. It works really good for the last 3 months. Resource usage is a lot lower than on Thanos.

Both solutions have their own Slack channels with developers and users there, so it is easy to get help and resolve issues.

In the end, I think I'll go with VM in my case, since it has less moving parts, doesn't need S3 backend (we are on-prem and don't have a production S3 storage) and lower resource usage. It can also ingest InfluxDB metrics, which is a massive bonus for me, since NOC team is using a solution that can only send metrics to InfluxDB (snmpcollector).

Re: Big Prometheus: Thanos, Cortex, M3DB and VictoriaMetrics at Scale

#22
post #16
post #13

Earlier quoted context omitted.

Prometheus author here (Julius). Generally agreed that you can get far with a single Prometheus server (or many independent vanilla Prometheus servers, potentially also using Prometheus's own federation). But I still recommend Thanos as an extension to a lot of people. I like Thanos because it's so easy to deploy alongside an existing Prometheus installation, while itself being mostly stateless (long-term state is ke…

Do you have by any chance recorded this?

No, sorry, it was a private commercial training.
Post reply on HN