Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CNCF's Cortex v1.0: scalable, fast Prometheus implementation

grafana.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#31

Post by ecnahc515 »

Florin_Andrei wrote: post #26
nopzor wrote: post #25

awesome job by the cortex team! there's a lot of good questions, and some confusion in this thread. here is my view. note: i'm definitely biased; am the co-founder/ceo at grafana labs. - at grafana labs we are huge fans of prometheus. it has become the most popular metrics backend for grafana. we view cortex and prometheus as complementary. we are also very active contributors to the prometheus project itself. in fac…

I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?

Thanos is probably one of the other popular choices. It's being heavily used in production by a number of companies, but I don't think they've branded it at "Prod ready" in a 1.0 release though.

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#32

Post by sciurus »

ecnahc515 wrote: post #31
Florin_Andrei wrote: post #26

Earlier quoted context omitted.

I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?

Thanos is probably one of the other popular choices. It's being heavily used in production by a number of companies, but I don't think they've branded it at "Prod ready" in a 1.0 release though.

Thanos doesn't have production support for local storage either. The only stable storage providers for it are google, amazon, and azure's object stores.

https://thanos.io/storage.md/

Interestingly, it looks like Cortex's support for local storage and object stores comes from using Thanos's storage engine. So once it's production ready in Thanos it will probably be production-ready in Cortex shortly thereafter.

https://cortexmetrics.io/docs/operations/blocks-storage/

I think for Cortex your safest storage options now are Bigtable, DynamoDB, or Cassandra.

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#33

Post by ecnahc515 »

outworlder wrote: post #24
ones_and_zeros wrote: post #10

Isn't prometheus an implementation and not an interface? I have "prometheus" running in my cluster, if it's not cortex, what implementation am I using?

You are using Prometheus. However, Prometheus can use different storage backends. The TSDB that it comes with is horrible. I mean, it's workable. And can store an impressive amount of data points. If you don't care about historical data or scale, it may be all you need. However, if your scale is really large, or if you care about the data, it may not be the right solution, and you'll need something like Cortex. For i…

The TSDB it uses is actually pretty state of the art. I think your pain point is more that it's designed for being used on local disk, but that doesn't mean it isn't possible to store the TSDB remotely. In fact, this is exactly how Thanos works.

The docs say Prometheus is not intended for long term storage because without a remote_write configuration, all data is persisted locally, and thus you will eventually hit limits on the amount that can be stored and queried locally. However, that is a limitation on how Prometheus is designed, not how the TSDB is designed, and which can be overcome by using a remote_write adapter.

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#34

Post by simonrobb »

gouthamve wrote: post #30
Florin_Andrei wrote: post #26

Earlier quoted context omitted.

I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?

The only one I know with "non-experimental" local-storage is VictoriaMetrics. But the big thing there is that data in VM is not replicated, so when you lose a disk/node, you lose that data. Having said that, both Thanos and Cortex have experimental local-storage modes that are pretty good. You could also try them for now while they get production ready.

M3 provides local storage but is not experimental, on top of that with cluster replication which VictoriaMetrics does not provide, and has a kubernetes operator to help scale out a cluster.

Disclosure: I work on the TSDB underlying M3 (M3DB) at Uber. Still worth checking out though!

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#35

Post by prungta »

Florin_Andrei wrote: post #26
nopzor wrote: post #25

awesome job by the cortex team! there's a lot of good questions, and some confusion in this thread. here is my view. note: i'm definitely biased; am the co-founder/ceo at grafana labs. - at grafana labs we are huge fans of prometheus. it has become the most popular metrics backend for grafana. we view cortex and prometheus as complementary. we are also very active contributors to the prometheus project itself. in fac…

I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?

I suggest checking out M3DB[1]. My team & I use it to serve metrics for all of Uber, we have ~1500 hosts across various clusters. It's serving us quite well.

[1]: https://github.com/m3db/m3

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#36

Post by Florin_Andrei »

gouthamve wrote: post #30
Florin_Andrei wrote: post #26

Earlier quoted context omitted.

I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?

The only one I know with "non-experimental" local-storage is VictoriaMetrics. But the big thing there is that data in VM is not replicated, so when you lose a disk/node, you lose that data. Having said that, both Thanos and Cortex have experimental local-storage modes that are pretty good. You could also try them for now while they get production ready.

> data in VM is not replicated, so when you lose a disk/node, you lose that data

The vmstorage component in VictoriaMetrics Server - is it RAID0-like (stripping) or RAID1-like (mirroring)?

https://github.com/VictoriaMetrics/VictoriaMetrics/tree/clus...

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#38

Post by ecnahc515 »

sciurus wrote: post #32
ecnahc515 wrote: post #31

Earlier quoted context omitted.

Thanos is probably one of the other popular choices. It's being heavily used in production by a number of companies, but I don't think they've branded it at "Prod ready" in a 1.0 release though.

Thanos doesn't have production support for local storage either. The only stable storage providers for it are google, amazon, and azure's object stores. https://thanos.io/storage.md/ Interestingly, it looks like Cortex's support for local storage and object stores comes from using Thanos's storage engine. So once it's production ready in Thanos it will probably be production-ready in Cortex shortly thereafter. https:…

I may have misinterpreted what they meant by local storage! I was reading that as having a local copy of the TSDB available to Prometheus, (eg: how Thanos works) versus Cortex which doesn't store metrics locally (IIRC).

What you said is correct and makes sense. Though, I would suspect either choice works with any S3 compatible API that can run on local storage, but I know that isn't necessarily what's meant by "local storage".

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#39

Post by sagichmal »

outworlder wrote: post #24
ones_and_zeros wrote: post #10

Isn't prometheus an implementation and not an interface? I have "prometheus" running in my cluster, if it's not cortex, what implementation am I using?

You are using Prometheus. However, Prometheus can use different storage backends. The TSDB that it comes with is horrible. I mean, it's workable. And can store an impressive amount of data points. If you don't care about historical data or scale, it may be all you need. However, if your scale is really large, or if you care about the data, it may not be the right solution, and you'll need something like Cortex. For i…

> The TSDB that it comes with is horrible.

The TSDB in Prometheus since 2.0 is excellent for its use case.

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#40

Post by zytek »

Congrats to Grafana Team!

If you're looking at scaling your Prometheus setup - check out also Victoria Metrics.

Operational simplicity and scalability/robustness are what drive me to it.

I used to to send metrics from multiple Kubernetes clusters with Prometheus - each cluster having Prom with remote_write directive to send metrics to central VictoriaMetrics service.

That way my "edge" prometheus installations are practically "stateless", easily set up using prometheus-operator. You don't even need to add persistent storage to them.

Post reply on HN