41–50 of 133 posts
Earlier quoted context omitted.
You can set up dist eval similar to how it was done in borgmon but you gotta do it manually (or maybe write an operator to automate). One of Monarchs core ideas is to do that behind the scenes for you
Prometheus' own docs say that distributed evaluation is "deemed infeasible".
Earlier quoted context omitted.
Is it really that different from, say, the way Prometheus supports histogram-based quantiles? https://prometheus.io/docs/practices/histograms/ Granted, it looks like Monarch supports a more cleanly-defined schema for distributions, whereas Prometheus just relies on you to define the buckets yourself and follow the convention of using a "le" label to expose them. But the underlying representation (an empirical CDF) se…
Much different. When you are reporting histograms you can combine them and see the true p50 or whatever across all the individual systems reporting the metric.
Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?
Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
A lot of Google projects seem to rely on other Google projects. In this case Monarch relies on spanner. I guess its nice to publish at least the conceptual design so that others can implement it in “rest of the world” case. Working with OSS can be painful, slow and time consuming so this seems like a reasonable middle ground (although selfishly I do wish all of this was source available).
I don't think there's any spanner necessity and iirc monarch existed pre-spanner.
A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.
In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…
Disclaimer: I am a co-author.
Earlier quoted context omitted.
Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
Prometheus is highly scalable?? What are you talking about??
It basically does the opposite of what every scalable system does.
To get HA you double you’re number of pollers.
To get scale your queries you aggregate them into other prometheii.
If this is scalability: everything is scalable.
A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.
In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…
Leaving the world of single numeric type for each datum will influence the next generation of open source metrics db.
Earlier quoted context omitted.
Can you elaborate? I’ve ran Prometheus at some scale and it’s performed fine.
You pretty quickly exceed what one instance can handle for memory, cpu or both. At that point you don't have any real good options to scale while maintaining a flat namespace (you need to partition).
Earlier quoted context omitted.
Prometheus is highly scalable?? What are you talking about??
It is not. It basically does the opposite of what every scalable system does. To get HA you double you’re number of pollers. To get scale your queries you aggregate them into other prometheii. If this is scalability: everything is scalable.
High Availability always requires duplication of effort. Scaling queries always requires sharding and aggregation at some level.
I've deployed stock Prometheus at global scale, O(100k) targets, with great success. You have to understand and buy into Prometheus' architectural model, of course.
Earlier quoted context omitted.
Prometheus' own docs say that distributed evaluation is "deemed infeasible".
Thats just like.. you know.. their opinion, man. https://prometheus.io/docs/prometheus/latest/federation/
> Federation allows a Prometheus server to scrape selected time series from another Prometheus server