Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CNCF's Cortex v1.0: scalable, fast Prometheus implementation

grafana.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#41

Post by Florin_Andrei »

ecnahc515 wrote: post #38
sciurus wrote: post #32

Earlier quoted context omitted.

Thanos doesn't have production support for local storage either. The only stable storage providers for it are google, amazon, and azure's object stores. https://thanos.io/storage.md/ Interestingly, it looks like Cortex's support for local storage and object stores comes from using Thanos's storage engine. So once it's production ready in Thanos it will probably be production-ready in Cortex shortly thereafter. https:…

I may have misinterpreted what they meant by local storage! I was reading that as having a local copy of the TSDB available to Prometheus, (eg: how Thanos works) versus Cortex which doesn't store metrics locally (IIRC). What you said is correct and makes sense. Though, I would suspect either choice works with any S3 compatible API that can run on local storage, but I know that isn't necessarily what's meant by "local…

"local storage" = I don't want to install yet another gizmo just to store data, nor do I want to use an external service for that

Batteries included.

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#42

Post by mmcclellan »

New to Cortex but when looking at a comparison of Prometheus and InfluxDB (like https://prometheus.io/docs/introduction/comparison/#promethe...) it appears that Cortex offers similar horizontal scalability features to the InfluxDB Enterprise offering. The linked comparison does note the difference between event logging and metrics recording but I am curious (choosy beggar that I am) whether others consider them separate tooling or whether it is possible to remain performant using one solution.

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#44

Post by valyala »

Florin_Andrei wrote: post #36
gouthamve wrote: post #30

Earlier quoted context omitted.

The only one I know with "non-experimental" local-storage is VictoriaMetrics. But the big thing there is that data in VM is not replicated, so when you lose a disk/node, you lose that data. Having said that, both Thanos and Cortex have experimental local-storage modes that are pretty good. You could also try them for now while they get production ready.

> data in VM is not replicated, so when you lose a disk/node, you lose that data The vmstorage component in VictoriaMetrics Server - is it RAID0-like (stripping) or RAID1-like (mirroring)? https://github.com/VictoriaMetrics/VictoriaMetrics/tree/clus...

It is more like RAID0. VictoriaMetrics shards time series among available vmstorage nodes. I.e. each vmstorage node contains a part of data stored in the cluster. This is usually named shared nothing architecture [1].

As for data replication, VictoriaMetrics offloads this task to the underlying storage, since the replication is hard to make properly [2]. Proper replication must be able to perform the following tasks additionally to copying the data to multiple nodes:

* To heal the data (aka to return back replication factor) after a node becomes permanently unavailable. The healing process mustn't degrade cluster performance and it must properly handle other cases mentioned below.

* To gracefully handle temporary unavailability of nodes.

* To survive network partitioning when nodes are temporarily split into multiple isolated subnetworks.

* To handle data corruption.

* To continue accepting new data at normal rate when a part of nodes are unavailable.

* To continue serving incoming requests with acceptable latency when a part of nodes are unavailable.

* To replicate data among multiple availability zones (AZ), so the cluster should continue accepting new data and serving requests if a single AZ becomes unavailable.

I'm unsure whether popular systems that claim replication support can handle all the cases mentioned above. The only system that seems to handle these cases properly is GCP persistent disks based on Colossus storage [3]. That's why it is recommended storing VictoriaMetrics data on GCP persistent disks.

[1] https://en.wikipedia.org/wiki/Shared-nothing_architecture

[2] https://github.com/VictoriaMetrics/VictoriaMetrics/tree/clus...

[3] https://medium.com/google-cloud/persistent-disks-and-replica...

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#45

Post by valyala »

number101010 wrote: post #6
netingle wrote: post #2

Hi! Tom, one of the Cortex authors here. Super proud of the team and this release - let me know if you have any questions!

Hey Tom! Can you outline how Cortex differs from some of the other available Prometheus backends?

VictoriaMetrics FAQ contains comparisons to Cortex [1] and Thanos [2].

[1] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#...

[2] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#...

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#46

Post by valyala »

netingle wrote: post #23
Rapzid wrote: post #22

Dat architecture tho: https://cortexmetrics.io/docs/architecture/ . Holy bi-gebus.

Thats the "microservices" mode - you can run it as a single process and the architecture becomes super boring. Its like looking at the module interdependencies of reasonably large piece of software; of course its going to look complicated.

According to Cortex docs [1], a single-process Cortex isn't production ready. It is intended for development and testing only.

[1] https://cortexmetrics.io/docs/configuration/single-process-c...

Re: CNCF's Cortex v1.0: scalable, fast Prometheus implementation

#47

Post by valyala »

Florin_Andrei wrote: post #36
gouthamve wrote: post #30

Earlier quoted context omitted.

The only one I know with "non-experimental" local-storage is VictoriaMetrics. But the big thing there is that data in VM is not replicated, so when you lose a disk/node, you lose that data. Having said that, both Thanos and Cortex have experimental local-storage modes that are pretty good. You could also try them for now while they get production ready.

> data in VM is not replicated, so when you lose a disk/node, you lose that data The vmstorage component in VictoriaMetrics Server - is it RAID0-like (stripping) or RAID1-like (mirroring)? https://github.com/VictoriaMetrics/VictoriaMetrics/tree/clus...

It is easy to implement RAID1-like replication in VictoriaMetrics: just set up independent VictoriaMetrics instances (single-node or clusters) and replicate all the incoming data simultaneously to these instances. This can be done either via providing multiple `remote_write->url` values in Prometheus configs or via providing multiple `-remoteWrite.url` command-line flags in vmagent [1]. Then query multiple VictoriaMetrics replicas via Promxy [2].

[1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...

[2] https://github.com/jacksontj/promxy

Post reply on HN