Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monarch: Google’s Planet-Scale In-Memory Time Series Database

micahlerner.com

Post reply on HN
Print view View original

31–40 of 133 posts

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#31

Post by [deleted] »

hn_go_brrrrr wrote: post #28
kasey_junk wrote: post #25

A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…

[deleted]

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#33

Post by jeffbee »

dilyevsky wrote: post #23
jeffbee wrote: post #10

Earlier quoted context omitted.

Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.

You can set up dist eval similar to how it was done in borgmon but you gotta do it manually (or maybe write an operator to automate). One of Monarchs core ideas is to do that behind the scenes for you

Prometheus' own docs say that distributed evaluation is "deemed infeasible".

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#34

Post by atdt »

dilyevsky wrote: post #21
candiddevmike wrote: post #9

Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?

It was originally push but i think they went back to sort of scheduled pull mode after a few years. There was a very in depth review doc written about this internally which maybe will get published some day

What's the go/ link?

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#35

Post by spullara »

kasey_junk wrote: post #25

A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.

Wavefront also has histogram ingestion (I wrote the original implementation, I'm sure it is much better now). Hugely important if you ask me but honestly I don't think that many customers use it.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#36

Post by spullara »

teraflop wrote: post #29
kasey_junk wrote: post #25

A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.

Is it really that different from, say, the way Prometheus supports histogram-based quantiles? https://prometheus.io/docs/practices/histograms/ Granted, it looks like Monarch supports a more cleanly-defined schema for distributions, whereas Prometheus just relies on you to define the buckets yourself and follow the convention of using a "le" label to expose them. But the underlying representation (an empirical CDF) se…

Much different. When you are reporting histograms you can combine them and see the true p50 or whatever across all the individual systems reporting the metric.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#37

Post by dilyevsky »

atdt wrote: post #34
dilyevsky wrote: post #21

Earlier quoted context omitted.

It was originally push but i think they went back to sort of scheduled pull mode after a few years. There was a very in depth review doc written about this internally which maybe will get published some day

What's the go/ link?

Can’t remember - just search on moma /s

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#40

Post by [deleted] »

spullara wrote: post #36
teraflop wrote: post #29

Earlier quoted context omitted.

Is it really that different from, say, the way Prometheus supports histogram-based quantiles? https://prometheus.io/docs/practices/histograms/ Granted, it looks like Monarch supports a more cleanly-defined schema for distributions, whereas Prometheus just relies on you to define the buckets yourself and follow the convention of using a "le" label to expose them. But the underlying representation (an empirical CDF) se…

Much different. When you are reporting histograms you can combine them and see the true p50 or whatever across all the individual systems reporting the metric.

[deleted]
Post reply on HN