Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monarch: Google’s Planet-Scale In-Memory Time Series Database

micahlerner.com

Post reply on HN
Print view View original

71–80 of 133 posts

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#71

Post by spullara »

nvarsj wrote: post #69
spullara wrote: post #36

Earlier quoted context omitted.

Much different. When you are reporting histograms you can combine them and see the true p50 or whatever across all the individual systems reporting the metric.

Can you elaborate a bit? You can do the same in Prometheus by summing the bucket counts. Not sure what you mean by “true p50” either. With buckets it’s always an approximation based on the bucket widths.

Ah, I misunderstood what you meant. If you are reporting static buckets I get how that is better than what folks typically do but how do you know the buckets a priori? Others back their histograms with things like https://github.com/tdunning/t-digest. It is pretty powerful as the buckets are dynamic based on the data and histograms can be added together.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#72

Post by gttalbot »

preseinger wrote: post #68
gttalbot wrote: post #66

Earlier quoted context omitted.

This means each new query over a certain size becomes a federation problem, so the friction for trying new things becomes very high above the scale of a single instance. Monitoring as a service has a lot of advantages.

Well you obviously don't issue metrics queries over arbitrarily large datasets, right? The Prometheus architecture reflects this invariant. You constrain queries against both time and domain boundaries.

Also, very large ad-hoc queries are supported, with really good user isolation, so that (in general) users don't harm each other.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#73

Post by preseinger »

dijit wrote: post #67
preseinger wrote: post #62

Earlier quoted context omitted.

Scalability isn't a well-defined term, and Prometheus isn't a database. :shrug:

Wrong on both counts Sorry for being rude, but this level of ignorance is extremely frustrating.

Ignorance? I'm a core Prometheus contributor and a 20+ year distsys veteran. Prometheus is not a database, and scalability is not well-defined. These are not controversial statements.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#74

Post by jeffbee »

sujayakar wrote: post #60
kasey_junk wrote: post #25

A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.

I've been pretty happy with datadog's distribution type [1] that uses their own approximate histogram data structure [2]. I haven't evaluated their error bounds deeply in production yet, but I haven't had to tune any bucketing. The linked paper [3] claims a fixed percentage of relative error per percentile. [1] https://docs.datadoghq.com/metrics/distributions/ [2] https://www.datadoghq.com/blog/engineering/computing-…

That is a very different tradeoff, though. A DDSketch is absolutely gigantic compared to a power-of-four binned distribution that could be implemented as a vector of integers. A practical DDSketch will be 5KiB+. And when they say DDSketch merges are "fast" they are comparing to other sketches that take microseconds or more to merge, not to CDF vectors that can be merged literally in nanoseconds.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#75

Post by buro9 »

deepsun wrote: post #63
buro9 wrote: post #13

Earlier quoted context omitted.

That's what Mimir solves

How does it compare to VictoriaMetrics?

100% Prometheus compatible, proven to scale to 1B active series.

It's not about comparisons, every tool has it's own place and feature set that may be right for you depending on what you're doing. But if you've reached the end of the road with Prometheus due to scale and you need massive scale and perfect compatibility... Then Mimir stands out.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#76

Post by dijit »

preseinger wrote: post #73
dijit wrote: post #67

Earlier quoted context omitted.

Wrong on both counts Sorry for being rude, but this level of ignorance is extremely frustrating.

Ignorance? I'm a core Prometheus contributor and a 20+ year distsys veteran. Prometheus is not a database, and scalability is not well-defined. These are not controversial statements.

It is extremely unbecoming to lie about who you are on this forum.

Scalability is defined differently depending on context; in this context (a monitoring/time series solution) it is defined as being able to hold a dataset larger than a single machine that scales horizontally.

Downsampling the data or transforming it does not meet that criteria, since that’s no longer the original data.

The way Prometheus “scales” today is a bolt-on passthrough with federation. It’s not designed for it at all, and means that your query will use other nodes as data sources until it runs out of ram evaluating the query. Or not.

The most common method of “scaling” Prometheus is making a tree; you can do that with anything (so it is not inherent to the technology, thus not a defining characteristic, if everything can be defined the same way then nothing can be- the term ceases to have meaning: https://valyala.medium.com/measuring-vertical-scalability-fo...)

I’ll tell you how influx scales: your data is horizontally sharded across nodes, queries are conducted cross shards.

That’s what scalability of the database layer is.

Not fetching data from other nodes and putting it together yourself.

Rehydrating from many datasets is not the storage system scaling: the collector layer doing the hydration is the thing that is scaling.

If you sold me a solution that used Prometheus underneath but was distributed across all nodes, perhaps we could talk.

But scalability is not a nebulous concept.

You should refer to your own docs if you think Prometheus isn’t a database, it certainly contains one: https://prometheus.io/docs/prometheus/latest/storage/

I should add (and extremely frustratedly): if you’re not lying and you’re a core Prometheus maintainer, you should know this. I’m deeply embarrassed to be telling you this.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#78

Post by Jedd »

lokar wrote: post #16
candiddevmike wrote: post #9

Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?

It's sort of a pull/push hybrid. The client connects to the collection system and is told how often to send each metric (or group of them) back over that same connection. You configure per target/metric collection policy centrally.

So, much like a Zabbix agent, with both active (push) & passive (pull) capabilities.

We're diving into OTEL, and the registration / discovery challenges don't seem to have any kind of best-practice consensus out there. We're looking at NodeRED (telegraf agent can query from same at startup) but it brings its own challenges.

I haven't read the full paper, but do you know if the push model was revisited mostly for auto-registration / discovery, or performance bottlenecks at the server, or some other concern?

Typically for us, once we've got the hard part - an entity registered - we're happy with pull only. A no-response from a prod end-point is an automatic critical. I guess at their scale there's more nuance around one or more agents being non-responsive.

EDIT: Oh, there's not much in the paper on the subject, as it happens. And yes, it's vanilla discovery woes.

"Push-based data collection improves system robustness while simplifying system architecture. Early versions of Monarch discovered monitored entities and “pulled” monitoring data by querying the monitored entity.

"This required setting up discovery services and proxies, complicating system architecture and negatively impacting overall scalability. Push-based collection, where entities simply send their data to Monarch, eliminates these dependencies."

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#79

Post by lokar »

Jedd wrote: post #78
lokar wrote: post #16

Earlier quoted context omitted.

It's sort of a pull/push hybrid. The client connects to the collection system and is told how often to send each metric (or group of them) back over that same connection. You configure per target/metric collection policy centrally.

So, much like a Zabbix agent, with both active (push) & passive (pull) capabilities. We're diving into OTEL, and the registration / discovery challenges don't seem to have any kind of best-practice consensus out there. We're looking at NodeRED (telegraf agent can query from same at startup) but it brings its own challenges. I haven't read the full paper, but do you know if the push model was revisited mostly for auto…

Push can help a bit, but you still have to know which endpoints you expect to hear from (if you want to detect they are missing).

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#80

Post by teraflop »

spullara wrote: post #36
teraflop wrote: post #29

Earlier quoted context omitted.

Is it really that different from, say, the way Prometheus supports histogram-based quantiles? https://prometheus.io/docs/practices/histograms/ Granted, it looks like Monarch supports a more cleanly-defined schema for distributions, whereas Prometheus just relies on you to define the buckets yourself and follow the convention of using a "le" label to expose them. But the underlying representation (an empirical CDF) se…

Much different. When you are reporting histograms you can combine them and see the true p50 or whatever across all the individual systems reporting the metric.

That is also possible in Prometheus, which is why I made the comparison.
Post reply on HN