Amazon Managed Service for Prometheus • Hacker News Board

Re: Amazon Managed Service for Prometheus

#71

Post by bboreham » Wed, Dec 16, 2020, 10:00 AM UTC

hagen1778 wrote: post #43

bboreham wrote: post #34

Earlier quoted context omitted.

It’s Cortex, though the particular configuration shares a lot of code with Thanos. (I am a Cortex maintainer)

If you know technical details, are there any metrics cardinality limitations?

There are soft limits _everywhere_, to stop people shooting themselves in the foot. Those can be raised by admins after checking the user knows what they are doing.

I do not know what the practical limits are right now; especially I do not know what size hardware AWS run it on.

If you were to search the Cortex Slack you would find people talking about instances with 100 million series, also people talking about work to improve scalability.

Re: Amazon Managed Service for Prometheus

#72

Post by potamic » Wed, Dec 16, 2020, 5:11 PM UTC

slyall wrote: post #37

The pricing just for the ingest seems way off. $0.002 for 10,000 metrics might not seem like much by even a simple node_exporter will grab 700 metrics every 15 seconds. Thats $24/month just to ingest the cpu/ram/diskspace data from each server. Plus storage and query costs. At work I have a single r4.xlarge instance handling 1.3 million metrics every 15 seconds. Storage is not clustered but cost is only $500/month. I…

That's probably lesser than your team's payroll budget :) Their positioning is that you can reduce the staff needed to operate and maintain these instances.

Re: Amazon Managed Service for Prometheus

#73

Post by valyala » Wed, Dec 16, 2020, 7:17 PM UTC

vishuk wrote: post #24

Do we know which scalable prometheus backend are they running? Chronosphere? Thanos?

They run Cortex.

Cortex architecture looks a bit over-engineered [1] when compared to other backends for Prometheus [2]. This may negatively affect system reliability while increasing operational costs and infrastructure costs. I hope Cortex architecture will be simplified in the future.

[1] https://github.com/cortexproject/cortex/blob/master/docs/arc...

[2] https://victoriametrics.github.io/Cluster-VictoriaMetrics.ht...

Re: Amazon Managed Service for Prometheus

#74

Post by heliodor » Thu, Dec 17, 2020, 4:00 AM UTC

jrv wrote: post #50

shitloadofbooks wrote: post #48

Earlier quoted context omitted.

Victoria Metrics is an absolutely superb drop in replacement.

It's not a drop-in replacement (even though it tries to sell itself as such), it's incompatible in a significant number of ways and throws away part of your data.

They made some pragmatic optimizations by dropping part of the data where it mostly doesn't matter. Who's actually affected by this? 0.01% of users?

Re: Amazon Managed Service for Prometheus

#75

Post by heliodor » Thu, Dec 17, 2020, 4:06 AM UTC

codeduck wrote: post #67

dilyevsky wrote: post #52

Earlier quoted context omitted.

What is in your opinion a better open source alternative to prometheus? Borgmon was inspiration for prometheus but was a totally different project so it is a complete rewrite

I feel like a broken record, but we are having great success with Victoria metrics as a drop in replacement.

Promscale looks interesting. Keep the architecture of Prometheus while storing the data in TimescaleDB and using SQL as the query language (together with the TimescaleDB-specific extensions to it). Does anyone actually like PromQL?

Re: Amazon Managed Service for Prometheus

#76

Post by bboreham » Thu, Dec 17, 2020, 8:07 AM UTC

AYBABTME wrote: post #55

I wonder how AWS is supporting the development of Prometheus. Are they financing the OSS developers who are spending countless hours dedicated to the project?

AWS is an investor in Weaveworks where the implementation (Cortex) was first created. Weaveworks had two Prometheus maintainers on staff at the time.

In the announcement it says AWS have a commercial relationship with Grafana Labs, where several Prometheus maintainers, community managers, etc. currently work.

(I work for Weaveworks)

Re: Amazon Managed Service for Prometheus

#77

Post by pointestimate » Wed, Dec 23, 2020, 2:38 AM UTC

jrv wrote: post #50

shitloadofbooks wrote: post #48

Earlier quoted context omitted.

Victoria Metrics is an absolutely superb drop in replacement.

It's not a drop-in replacement (even though it tries to sell itself as such), it's incompatible in a significant number of ways and throws away part of your data.

No, it's better. You can focus on the thing you're measuring rather than the bloody platform.