What is the scale out process with Prometheus? Is sharding/replication a manual setup process or is it automated? What's involved in scaling out?
Prometheus reaches 1.0
21–30 of 60 posts
Re: Prometheus reaches 1.0
#22Prometheus cofounder here - we're happy to take any questions. Huge congrats to everyone who made this release possible and for all the excellent work over the years that lead up to this!
Is it even right to try to compare this with New Relic? Prometheus looks very interesting and I'm trying to figure out if this is something useful to us. Thanks!
Not quite, though there is some overlap. Someone else asked me to compare Prometheus to other tools in the APM (application performance monitoring) space, and I'm going to share the summary I came up with:
The way I would describe Prometheus in relation to those other tools:
- Prometheus is open-source and self-hosted.
- Prometheus is about dimensional numeric time series metrics only (no log-based analysis, no per-request tracing, etc.).
- Prometheus has a strong focus on systems and service monitoring, not so much on business metrics.
- Prometheus is more of a Swiss army knife of monitoring rather than a ready-to-drop-in package that starts monitoring everything automatically.
- Prometheus is very much about whitebox monitoring and manually defining any metrics that could be useful for you (although we support blackbox exporting and bridging metrics from existing systems as well).
- We don't do machine-learning-style anomaly detection, but we do alerting based on manually defined rules.
- For a purely metrics-based solution, the insight we deliver is one of the best in the field (via the dimensional data model and the query language to go with it).
- Many open-source projects are starting to expose native Prometheus metrics (like k8s, etcd, ...), which gives Prometheus an advantage when being used together with those.
EDIT: Also try the "Getting Started" tutorial - that should only take a couple of minutes to try it out: https://prometheus.io/docs/introduction/getting_started/
Re: Prometheus reaches 1.0
#23What is the scale out process with Prometheus? Is sharding/replication a manual setup process or is it automated? What's involved in scaling out?
http://www.robustperception.io/scaling-and-federating-promet... explains it. Unless you're absolutely massive, it's fairly easy.
Re: Prometheus reaches 1.0
#24What is the scale out process with Prometheus? Is sharding/replication a manual setup process or is it automated? What's involved in scaling out?
Usually you start by some functional sharding (giving each team of services their own Prometheus servers), but also having per-datacenter Prometheus servers and then some hierarchical federation layer ontop of that. There's no built-in automatic horizontal scaling though (which would be going against the design goal of not having a clustered system, for reliability).
Some resources:
- Scaling in general: http://www.robustperception.io/scaling-and-federating-promet...
- Federation: https://prometheus.io/docs/operating/federation/
Re: Prometheus reaches 1.0
#25What is the best GitHub issue to follow to find out when Prometheus supports events older than the configurable 5 minute limit? I'm blocked because a network partition can mean no stats. Edit: Maybe https://github.com/prometheus/prometheus/issues/398 ? Also, limit is configurable.
That would be the correct issue about the staleness limit, yes. Note that Prometheus does not track individual events, but only numerical time series and their current and historical values. What's the exact use case you're stuck on? I guess you are using the pushgateway with client-side timestamps?
Yes, I have to collect metrics in a very restricted and separated production environment then ship them over to a completely separate system for reporting. My impression is that Prometheus just isn't the right fit.
Re: Prometheus reaches 1.0
#26Earlier quoted context omitted.
That would be the correct issue about the staleness limit, yes. Note that Prometheus does not track individual events, but only numerical time series and their current and historical values. What's the exact use case you're stuck on? I guess you are using the pushgateway with client-side timestamps?
Yes, I have to collect metrics in a very restricted and separated production environment then ship them over to a completely separate system for reporting. My impression is that Prometheus just isn't the right fit.
That depends - but yeah, those situations can be tricky sometimes.
If you are pushing metric states regularly though (more than every 5m) or don't set client-side timestamps, that usually works though. But maybe you have an even more special use case there regarding those metrics and the staleness?
Re: Prometheus reaches 1.0
#27I my have misread the documentation but there seems to be no way to get the same output as the highestAverage, highestCurrent, or highestCurrent functions from graphite. Not sure how to filter through say 300 servers and select out the top 10 for a particular time span. I would think that it would be a common need but I guess I'm missing something?
Prometheus range queries work a bit differently, so this is not 100% reproducible in a graph query, but a similar thing is possible. A given PromQL expression is evaluated at every resolution step along the graph and doesn't have context about what the "graph range" is. At every evaluation point, it can still look back over a given time window, but that's more of a sliding window approach then and independent of what…
I suppose what I'm asking for isn't really possible yet. Hopefully it ends up getting implemented at some point.
Still look forward to rolling out Prometheus for all of the other great features. Congrats on the release!
Re: Prometheus reaches 1.0
#28Earlier quoted context omitted.
Prometheus range queries work a bit differently, so this is not 100% reproducible in a graph query, but a similar thing is possible. A given PromQL expression is evaluated at every resolution step along the graph and doesn't have context about what the "graph range" is. At every evaluation point, it can still look back over a given time window, but that's more of a sliding window approach then and independent of what…
I suppose what I'm asking for isn't really possible yet. Hopefully it ends up getting implemented at some point. Still look forward to rolling out Prometheus for all of the other great features. Congrats on the release!
Thanks!
Re: Prometheus reaches 1.0
#29Is anyone using Promotheus for monitoring micro-services deployed with Kubernetes? Any feedbacks?
Re: Prometheus reaches 1.0
#30Is anyone using Promotheus for monitoring micro-services deployed with Kubernetes? Any feedbacks?
There's a few, here's a recent talk by Weave for example: http://www.slideshare.net/weaveworks/kubernetes-and-promethe...