Big Prometheus: Thanos, Cortex, M3DB and VictoriaMetrics at Scale

I enjoyed the post. Good links to a lot of relevant, recent stories & events.

Not the article's fault, but it cites the "ClickHouse Cost-Efficiency in Action: Analyzing 500 Billion Rows on an Intel NUC" article that was published January 1. It's a week old, & I kind of feel like I'm never going to get away with it. It seems like a great, fun, interesting premise, but the authors took what is a challenging, huge data-set, and, under the guise of making the data look "realistic" they drained all the entropy out of the dataset, & then claimed they were 10-100x faster.

Well, yes, maybe for some workloads maybe. Maybe the changes they made might in some circumstances be "realistic" for some IoT use cases, maybe.

But I feel like I'm going to see this article come up again, and again, and again. And each time, I'll have these frustrations, about how while they may still be running queries on the same number of rows, they are running queries on many orders of magnitude less data. It's a fun read, & genuinely useful- in some circumstances- tech, but I don't expect to see this nuance showing up. I'm already weary, seeing this Clickhouse article again.

Stop reading the articles, benchmark it yourself then write about it.

I think you're conflating the scraping of time-series data with it's storage.

Cortex author here (Tom Wilkie). Great post that honestly highlights the differences between these systems - thank you!

The biggest take home here - and the first thing the post mentions - is the a single HA pair of Prometheus servers is enough for 80-90% of people. TLDR you probably don’t need Cortex (or Thanos, etc)...

...unless you run multiple, segregated networks (regions). Then something like Thanos (or Cortex) is useful - not for a the scale argument, but because you need a way to “federate” queries and get that global view. IMO!

When my team agreed to use Prometheus from the client side we looked at Thanos, Cortex, and M3DB, but none of them gave us the flexibility and comfort of adoption for a small team providing a service to 10s of internal groups. We have many private internal DCs and needed metrics to be stored in the cloud, pulling data to the cloud seemed awkward and required access rights we couldn't get.

We ended up using Postgres 10 w/ TimeScaleDB and their Prometheus plugin with a simple emulated push gateway that converts a prom formatted http post to a postgres batch insert. Postgres is 3 nodes monitored with Patroni.

Working great for us and handling 1000+ metrics a second with ease and we get SQL for both real-time metrics for monitoring and analytics for business needs. We are using about 10-15% of our systems giving us room to grow.