21–30 of 133 posts
Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?
I broke this once several years ago. I even use the incident number in my random usernames to see if a Googler recognizes it.
Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?
Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
A lot of Google projects seem to rely on other Google projects. In this case Monarch relies on spanner. I guess its nice to publish at least the conceptual design so that others can implement it in “rest of the world” case. Working with OSS can be painful, slow and time consuming so this seems like a reasonable middle ground (although selfishly I do wish all of this was source available).
Spanner may be hard to set up even with source code available. It relies on atomic clocks for reliable ordering of events.
All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.
I don’t really grasp why this is a useful spot in the trade off space from a quick skim. Seems risky.
There’s a good talk on Monarch https://youtu.be/2mw12B7W7RI Why it exists is laid out quite plainly. The pain of it is we’re all jumping on Prometheus (borgmon) without considering why Monarch exists. Monarch doesn’t have a good corollary outside of google. Maybe some weird mix of timescale DB backed by cockroachdb with a Prometheus push gateway.
[1] https://news.ycombinator.com/item?id=16879392
Disclaimer: I work at vmware on an unrelated thing.
I broke this once several years ago. I even use the incident number in my random usernames to see if a Googler recognizes it.
A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.
A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.
Granted, it looks like Monarch supports a more cleanly-defined schema for distributions, whereas Prometheus just relies on you to define the buckets yourself and follow the convention of using a "le" label to expose them. But the underlying representation (an empirical CDF) seems to be the same, and so the accuracy tradeoffs should also be the same.
A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.
In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…