Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Managed Service for Prometheus

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 77 posts

Re: Amazon Managed Service for Prometheus

#41

Post by qz2 »

latchkey wrote: post #39
rfratto wrote: post #32

Earlier quoted context omitted.

One of the Loki maintainers here (though I mostly work on other stuff now). I promise it's not difficult on purpose. We've put a lot of effort into optimizing the Kubernetes experience that non-containerized installations haven't been getting as much attention. We'd be thrilled to have system packages for Loki that also set it up as a service, it's just not something we've been able to spend time doing ourselves yet.

It isn't just loki, but the whole stack. Grafana is the only project mentioned that has a debian installer. The expectation that someone doing greenfield development is going to jump into k8s just to use the software is kind of weird.

I’m deploying it (prom, alertmanager, pushgateway, grafana) on native hardware via ansible and it’s not difficult. Not Loki (yet). It’s all just go binaries you fire up with systemd with a single config file.

I find it harder to deploy reliably on kubernetes with persistent volumes etc.

Re: Amazon Managed Service for Prometheus

#42

Post by mchene »

alexhf wrote: post #27

I don't see any mention of Pushgateway. They'll need to add that or I won't be able to monitor ephemeral jobs.

Hey... Marc here from AWS. I'm the PM lead for this service. Thank you for the feedback. Pushgateway is important for our customers and it is a feature we are looking to support as part of our roadmap. For the time being, you can continue to use the Pushgateway as you do today and remote write the metrics to AMP for long term storage and querying!

Re: Amazon Managed Service for Prometheus

#43

Post by hagen1778 »

bboreham wrote: post #34
vishuk wrote: post #24

Do we know which scalable prometheus backend are they running? Chronosphere? Thanos?

It’s Cortex, though the particular configuration shares a lot of code with Thanos. (I am a Cortex maintainer)

If you know technical details, are there any metrics cardinality limitations?

Re: Amazon Managed Service for Prometheus

#45

Post by smcleod »

pram wrote: post #28
Thaxll wrote: post #21

Earlier quoted context omitted.

Prometheus is not easy to run at scale on the storage side.

This is all relative but I don't personally think so. Not on EC2+EBS, anyway. Certainly not as difficult as running/scaling an ES or Kafka cluster.

Out of interest what do you find hard about running ElasticSearch clusters?

In my experience ES has been one of the easiest clustered / highly available and sharded systems I've ever run - especially for how incredibly performant and reliable it is.

I've generally found that beyond right sizing your nodes, indexes and shard configuration - it pretty much just works without ever really having issues.

Re: Amazon Managed Service for Prometheus

#46

Post by throwaway343233 »

slyall wrote: post #37

The pricing just for the ingest seems way off. $0.002 for 10,000 metrics might not seem like much by even a simple node_exporter will grab 700 metrics every 15 seconds. Thats $24/month just to ingest the cpu/ram/diskspace data from each server. Plus storage and query costs. At work I have a single r4.xlarge instance handling 1.3 million metrics every 15 seconds. Storage is not clustered but cost is only $500/month. I…

Pricing makes sense if you consider how Amazon operates at this point.

You put basically a MVP product out there with abnormal pricing. Your enterprise customers that are drowning in money can start using it and using that money you can grow your org by hiring more engineers. At this point you start working on adding new features and do cost optimization. Since your whole architecture was designed based on "we have to ship this ASAP", you deliver some real nice cost reduction easily. Then you reflect this to your customers and gain goodwill and good PR.

Re: Amazon Managed Service for Prometheus

#47

Post by kobalsky »

slyall wrote: post #37

The pricing just for the ingest seems way off. $0.002 for 10,000 metrics might not seem like much by even a simple node_exporter will grab 700 metrics every 15 seconds. Thats $24/month just to ingest the cpu/ram/diskspace data from each server. Plus storage and query costs. At work I have a single r4.xlarge instance handling 1.3 million metrics every 15 seconds. Storage is not clustered but cost is only $500/month. I…

both google and amazon are insane with their observability services.

we ran away screaming from stackdriver when we saw how costs started piling up.

thank god for prometheus and grafana.

Re: Amazon Managed Service for Prometheus

#48

Post by shitloadofbooks »

Thaxll wrote: post #21
pram wrote: post #6

Yeah I dunno about this, and the grafana service. They’re not exactly complicated to run on their own. At this pricing you may as well be on Datadog.

Prometheus is not easy to run at scale on the storage side.

Victoria Metrics is an absolutely superb drop in replacement.

Re: Amazon Managed Service for Prometheus

#49

Post by jitl »

rfratto wrote: post #32
latchkey wrote: post #26

I just went through the "process" of installing Grafana, Loki, Promtail and Prometheus on an ubuntu box and it is almost like the company behind all of this has gone out of the their way to make it hard. It isn't really _that_ difficult to get set up, but it also isn't 'apt install' easy (you really want me to create my own startup scripts?) and required me to build my own documentation on how I installed everything.

One of the Loki maintainers here (though I mostly work on other stuff now). I promise it's not difficult on purpose. We've put a lot of effort into optimizing the Kubernetes experience that non-containerized installations haven't been getting as much attention. We'd be thrilled to have system packages for Loki that also set it up as a service, it's just not something we've been able to spend time doing ourselves yet.

Honestly I mostly throw out the Debian service definitions anyways - when clustering or interacting with Chef or Ansible or whatever, you end up building a lot of ‘smarts’ around a custom supervisor like Runit or skarnet or systemd

Re: Amazon Managed Service for Prometheus

#50

Post by jrv »

shitloadofbooks wrote: post #48
Thaxll wrote: post #21

Earlier quoted context omitted.

Prometheus is not easy to run at scale on the storage side.

Victoria Metrics is an absolutely superb drop in replacement.

It's not a drop-in replacement (even though it tries to sell itself as such), it's incompatible in a significant number of ways and throws away part of your data.
Post reply on HN