Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monarch: Google’s Planet-Scale In-Memory Time Series Database

micahlerner.com

Post reply on HN
Print view View original

121–130 of 133 posts

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#121

Post by heinrichhartman »

joeblubaugh wrote: post #114
heinrichhartman wrote: post #111

Earlier quoted context omitted.

There are a large amount of subtle tradeoffs around the bucketing scheme (log, vs. log-linear, base) and memory layout (sparse, dense, chunked) the amount of configurability in the histogram space (circllhist, DDSketch, HDRHistogram, ...). A good overview is this discussion here: https://github.com/open-telemetry/opentelemetry-specificatio... As for the circllhist: There are no knobs to turn. It uses base 10 and two…

Right - it’s why I said “in the abstract.” You could do it and still have a log-linear format. Base 10 works great for real-world distributions. Thanks for making and ossing circllhist. I’ve been close by to the whole “what’s the OTel histogram going to be” discussion for the last many months and learned a lot from that. That discussion is what introduced me to circllhist and got me using them.

Good to hear. I also learned a lot from this discussion!

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#122

Post by Jedd »

gttalbot wrote: post #115
Jedd wrote: post #103

Earlier quoted context omitted.

Again, a highly envious feature of a large organisation with almost exclusively bespoke applications that can port & integrate custom libraries directly into applications. Us little people have to contend with mostly black box applications, or occasionally native instrumented with at best a prom-alike endpoint. Amusingly in the pre-web 1990's, at Telstra (Australia telco) we also developed & implemented a custom perf…

Yeah that's a tough one, and it was indeed a luxury to be able to have a uniform instrumentation library.

Re-reading my comment, it may have been worded badly.

The phrase 'you aren't Google' is true for 99.9% of us. We all get to fix the problems in front of us, was my point. And at that scale you've got unique problems, but also an architecture, imperative, and most importantly an ethos that lets you solve them in this fashion.

I was more reflecting on the (actually pretty fine) tools available to SREs caring for off-the-shelf OS's and products, and a little on the whole 'we keep coming full circle' thing.

Anyway, I very much appreciate the insights.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#123

Post by hiptobecubic »

xyzzy_plugh wrote: post #97
preseinger wrote: post #68

Earlier quoted context omitted.

Well you obviously don't issue metrics queries over arbitrarily large datasets, right? The Prometheus architecture reflects this invariant. You constrain queries against both time and domain boundaries.

That's a choice Prometheus has made, not an invariant. Many scalable systems support arbitrarily large queries as they themselves scale. Prometheus pretty much prevents creating arbitrarily large datasets to begin with, so the point is kind of moot. Requiring query authors to understand the arrangement of their aggregation layer seems like a reasonable idea but is in fact quite ridiculous.

To be fair, monarch has scaling limits that require you to be aware of how aggregation is done as well. It's amazing what it can do, but depending on how much data you have you might need to design your schema/collection with monarch's architecture in mind.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#124

Post by hiptobecubic »

dilyevsky wrote: post #58
bpicolo wrote: post #50

Earlier quoted context omitted.

Prometheus federation isn't distributed evaluation. It "federates" from other nodes onto a single node. > Federation allows a Prometheus server to scrape selected time series from another Prometheus server

Collect directly from shard-level prometheus then aggregate using /federate at another level. That’s how thanos also works afaik

This still requires the query to fit on a single node at some point unless you're doing multi-level aggregation. Monarch does that. Being able to push expensive, highly-parallelizable aspects of the query down into the leaves is a massive benefit.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#125

Post by SuperQue »

mherdeg wrote: post #85
heinrichhartman wrote: post #45

Earlier quoted context omitted.

Circonus Histograms solve that by using a universal bucketing scheme. Details are explained in this paper: https://arxiv.org/abs/2001.06561 Disclaimer: I am a co-author.

Wow, this is a fantastic solution to some questions I've had rattling around in my head for years about the optimal bucket choices to minimize error given a particular set of buckets. Do I read right that circllhist has a pretty big number of bin sizes and is not configurable (except that they're sparse so may be small on disk)? I've found myself using high-cardinality Prometheus metrics where I can only afford 10-15…

Prometheus is in the process of developing a similar automatic log-linear histogram bucket type. The goal is to make it as cheap as a 10-15 bucket histogram, but not require pre-defined buckets.

https://github.com/prometheus/prometheus/tree/sparsehistogra...

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#126

Post by hiptobecubic »

foota wrote: post #99
preseinger wrote: post #82

Earlier quoted context omitted.

> in this context (a monitoring/time series solution) it is defined as being able to hold a dataset larger than a single machine that scales horizontally. This just isn't true :shrug: Horizontal scaling is one of many strategies.

I think the disconnect is that promethus helps a user to shard things, but it's not automatic. Other time series databases and monitoring solutions automatically distribute and query across servers. It's like postgres vs newswl (aka foundationdb, spanner, etc.,). While Prometheus supports sharding queries when a user sets it up, my understanding is that this has to be done manually, which is definitely less convenien…

Prometheus supports sharding queries the way a screwdriver supports turning multiple screws at once. You can design a system yourself that includes the screwdriver, which will turn all the screws, but there's nothing inherent to the screwdriver that helps you with this. If "scalability" just means "you can use it to design something new from scratch that scales" then the term is pretty meaningless.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#127

Post by kasey_junk »

SuperQue wrote: post #125
mherdeg wrote: post #85

Earlier quoted context omitted.

Wow, this is a fantastic solution to some questions I've had rattling around in my head for years about the optimal bucket choices to minimize error given a particular set of buckets. Do I read right that circllhist has a pretty big number of bin sizes and is not configurable (except that they're sparse so may be small on disk)? I've found myself using high-cardinality Prometheus metrics where I can only afford 10-15…

Prometheus is in the process of developing a similar automatic log-linear histogram bucket type. The goal is to make it as cheap as a 10-15 bucket histogram, but not require pre-defined buckets. https://github.com/prometheus/prometheus/tree/sparsehistogra...

Correct me if I’m wrong but I thought the sparse histogram effort in Prometheus will still use single metric line counters for its storage abstraction?

I think it’s a great addition to the product and will excitedly use it but it’s a pretty big difference from a histogram centric db like circonus’ or a schema’d one like monarch.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#128

Post by preseinger »

bboreham wrote: post #113
preseinger wrote: post #65

Earlier quoted context omitted.

It can? It just partitions the query over multiple nodes?

Where is the code to do that?

Oh, I see what you mean. Sure, it's in Thanos, or Grafana, or whatever layer above, not Prometheus itself.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#129

Post by fleventynine »

hn_go_brrrrr wrote: post #28
kasey_junk wrote: post #25

A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…

The histograms are useful on their own (visualized as a heatmap). If percentile lines are necessary (and they often aren't), I prefer to overlay them on top of the heatmap so it is clear where the bucket edges are.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#130

Post by hintymad »

dan-robertson wrote: post #119
hintymad wrote: post #108

I don't quite get the benefit of pull model by default either. A pull model by default means that it's not easy for a library to publish its metrics. For instance, every god damn application is expected to implement a `/metrics` endpoint for a freaking agent to publish the application's metrics to Prometheus. With Monarch, any library or application can simply publish metrics to Monarch's API. Similarly in Netflix, p…

The general theory is that if a push-based system is getting overloaded you drop metric submissions but if a pull-based system is overloaded it will query less frequently and you’ll just get less resolution.

Thanks. It's a valid thought on such trade-off. I do think we can still favor productivity without losing resolution, though, for the following two reasons:

1. A pull-based system can pull less when the system is overload, which means a pulled service needs to keep historical stats. For instance, the endpoint `/metric` needs to keep previous gauge values or the accumulated counters. That said, a push-based metric library can keep history too. Indeed, it is exactly what the micrometers library does.

2. Don't let the metric system overload. This sounds like a hyperbole, but it is what companies do in practice: telemetry system is so foundational and critical to an internet company that it should always run smoothly.

Post reply on HN