Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monarch: Google’s Planet-Scale In-Memory Time Series Database

micahlerner.com

Post reply on HN
Print view View original

111–120 of 133 posts

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#111

Post by heinrichhartman »

joeblubaugh wrote: post #102
mherdeg wrote: post #85

Earlier quoted context omitted.

Wow, this is a fantastic solution to some questions I've had rattling around in my head for years about the optimal bucket choices to minimize error given a particular set of buckets. Do I read right that circllhist has a pretty big number of bin sizes and is not configurable (except that they're sparse so may be small on disk)? I've found myself using high-cardinality Prometheus metrics where I can only afford 10-15…

I’ve used these log-linear history in a few pieces of code. There is some configurability in the abstract - you could choose a different logarithm base. In practice none of the implementations seem to provide that. Within the each set of buckets for a given log base you have reasonable precision at that magnitude. If your metric is oscillating around 1e6 you shouldn’t care much about the variance at 1e2, and with thi…

There are a large amount of subtle tradeoffs around the bucketing scheme (log, vs. log-linear, base) and memory layout (sparse, dense, chunked) the amount of configurability in the histogram space (circllhist, DDSketch, HDRHistogram, ...). A good overview is this discussion here:

https://github.com/open-telemetry/opentelemetry-specificatio...

As for the circllhist: There are no knobs to turn. It uses base 10 and two decimal digits of precision. In the last 8 years I have not seen a single use-case in the operational domain where this was not appropriate.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#112

Post by heinrichhartman »

NHQ wrote: post #101
heinrichhartman wrote: post #45

Earlier quoted context omitted.

Circonus Histograms solve that by using a universal bucketing scheme. Details are explained in this paper: https://arxiv.org/abs/2001.06561 Disclaimer: I am a co-author.

Is it lossy to store data this way?

Yes. But also MUCH cheaper. You loose (1) ordering (2) ~5% of precision of the individual measurements.

In practice, this still allows for highly precise percentile calculations (<0.1% error), see the evaluation in the Circllhist paper.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#113

Post by bboreham »

preseinger wrote: post #65
jeffbee wrote: post #57

Earlier quoted context omitted.

Prometheus cannot evaluate a query over time series that do not fit in the memory of a single node, therefore it is not scalable. The fact that it could theoretically ingest an infinite amount of data that it cannot thereafter query is not very interesting.

It can? It just partitions the query over multiple nodes?

Where is the code to do that?

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#114

Post by joeblubaugh »

heinrichhartman wrote: post #111
joeblubaugh wrote: post #102

Earlier quoted context omitted.

I’ve used these log-linear history in a few pieces of code. There is some configurability in the abstract - you could choose a different logarithm base. In practice none of the implementations seem to provide that. Within the each set of buckets for a given log base you have reasonable precision at that magnitude. If your metric is oscillating around 1e6 you shouldn’t care much about the variance at 1e2, and with thi…

There are a large amount of subtle tradeoffs around the bucketing scheme (log, vs. log-linear, base) and memory layout (sparse, dense, chunked) the amount of configurability in the histogram space (circllhist, DDSketch, HDRHistogram, ...). A good overview is this discussion here: https://github.com/open-telemetry/opentelemetry-specificatio... As for the circllhist: There are no knobs to turn. It uses base 10 and two…

Right - it’s why I said “in the abstract.” You could do it and still have a log-linear format. Base 10 works great for real-world distributions.

Thanks for making and ossing circllhist. I’ve been close by to the whole “what’s the OTel histogram going to be” discussion for the last many months and learned a lot from that. That discussion is what introduced me to circllhist and got me using them.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#115

Post by gttalbot »

Jedd wrote: post #103
gttalbot wrote: post #92

Earlier quoted context omitted.

And yeah, everyone used the same instrumentation library, which could be used in a lot of different contexts, sometimes surprising!

Again, a highly envious feature of a large organisation with almost exclusively bespoke applications that can port & integrate custom libraries directly into applications. Us little people have to contend with mostly black box applications, or occasionally native instrumented with at best a prom-alike endpoint. Amusingly in the pre-web 1990's, at Telstra (Australia telco) we also developed & implemented a custom perf…

Yeah that's a tough one, and it was indeed a luxury to be able to have a uniform instrumentation library.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#116

Post by gttalbot »

Too wrote: post #106
Jedd wrote: post #103

Earlier quoted context omitted.

Again, a highly envious feature of a large organisation with almost exclusively bespoke applications that can port & integrate custom libraries directly into applications. Us little people have to contend with mostly black box applications, or occasionally native instrumented with at best a prom-alike endpoint. Amusingly in the pre-web 1990's, at Telstra (Australia telco) we also developed & implemented a custom perf…

The rest of us can only hope Opentelemetry becomes more widely adopted. They have put in a lot of effort in decoupling the application instrumentation from the monitoring solution, to allow more rich instrumentation than just a prom-alike endpoint.

Yeah I hope OpenTelemetry works out as well.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#117

Post by dan-robertson »

heinrichhartman wrote: post #45
hn_go_brrrrr wrote: post #28

Earlier quoted context omitted.

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…

Circonus Histograms solve that by using a universal bucketing scheme. Details are explained in this paper: https://arxiv.org/abs/2001.06561 Disclaimer: I am a co-author.

You’ve a typo in def 3.20. Should be ‘losing’. And another in §3.4, should be ‘continuous’. I didn’t thoroughly look for more.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#118

Post by dan-robertson »

wbl wrote: post #95
praptak wrote: post #15

Earlier quoted context omitted.

Spanner may be hard to set up even with source code available. It relies on atomic clocks for reliable ordering of events.

Atomic clocks aren't that exotic, and a GPS disciplined ovenized quartz oscillator will do just fine outside of a disruption. The hard part is getting the right sampling semantics, requiring end to end error analysis.

It’s pretty hard to get well synchronised clocks between servers in a datacentre.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#119

Post by dan-robertson »

hintymad wrote: post #108

I don't quite get the benefit of pull model by default either. A pull model by default means that it's not easy for a library to publish its metrics. For instance, every god damn application is expected to implement a `/metrics` endpoint for a freaking agent to publish the application's metrics to Prometheus. With Monarch, any library or application can simply publish metrics to Monarch's API. Similarly in Netflix, p…

The general theory is that if a push-based system is getting overloaded you drop metric submissions but if a pull-based system is overloaded it will query less frequently and you’ll just get less resolution.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#120

Post by gttalbot »

ignoramous wrote: post #110
gttalbot wrote: post #51

Earlier quoted context omitted.

Pull collection eventually became a real scaling bottleneck for Monarch. The way the "pull" collection worked was that there was an external process-discovery mechanism, which the leaf used to connect to the entities it was monitoring, the leaf backend processes would connect to the monitored entities to an endpoint that the collection library would listen on, and those entities collection libraries would stream the…

Thanks. What are some problems (or peculiarities that otherwise didn't exist) with the push based setup? At another BigCloud, pull/push made for tasty design discussions as well, given the absurd scale of it all. General consensus was, smaller fleet always pulls from its downstream; push only if downstream and upstream both have similar scaling characteristics.

The system design choice was to make data visible to queries as soon as possible after being pushed to Monarch, to satisfy alerting guarantees.

Thus there was no queue like a pubsub or Kafka in front of Monarch.

At scale this required a "smoothness of flow". What I mean by this is that at the scale the system was operating the extent and shape of the latency long tail began to matter. If there are many many many many thousands of RPCs flowing through servers in the intermediate routing layers, any pauses at that layer or at the leaf layer below that extended even a few seconds could cause queueing problems at the routing layer that could impact flows to leaf instances that were not delayed. This would impact quality of collection.

Even something as simple as updating a range map table at the routing layer had to be done carefully to avoid contention during the update so as to not disturb the flow, which in practice could mean updating two copies of the data structure in a manner analogous to a blue green deployment.

At the leaf backends this required decoupling--to make eventual--many ancillary data structure updates for data structures that were consulted in the ingest path, and to eventually get to the point where queries and ingest shared no locks.

Post reply on HN