Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monarch: Google’s Planet-Scale In-Memory Time Series Database

micahlerner.com

Post reply on HN
Print view View original

101–110 of 133 posts

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#101

Post by NHQ »

heinrichhartman wrote: post #45
hn_go_brrrrr wrote: post #28

Earlier quoted context omitted.

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…

Circonus Histograms solve that by using a universal bucketing scheme. Details are explained in this paper: https://arxiv.org/abs/2001.06561 Disclaimer: I am a co-author.

Is it lossy to store data this way?

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#102

Post by joeblubaugh »

mherdeg wrote: post #85
heinrichhartman wrote: post #45

Earlier quoted context omitted.

Circonus Histograms solve that by using a universal bucketing scheme. Details are explained in this paper: https://arxiv.org/abs/2001.06561 Disclaimer: I am a co-author.

Wow, this is a fantastic solution to some questions I've had rattling around in my head for years about the optimal bucket choices to minimize error given a particular set of buckets. Do I read right that circllhist has a pretty big number of bin sizes and is not configurable (except that they're sparse so may be small on disk)? I've found myself using high-cardinality Prometheus metrics where I can only afford 10-15…

I’ve used these log-linear history in a few pieces of code. There is some configurability in the abstract - you could choose a different logarithm base.

In practice none of the implementations seem to provide that. Within the each set of buckets for a given log base you have reasonable precision at that magnitude. If your metric is oscillating around 1e6 you shouldn’t care much about the variance at 1e2, and with this scheme you don’t have to tune anything to provide for that.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#103

Post by Jedd »

gttalbot wrote: post #92
gttalbot wrote: post #91

Earlier quoted context omitted.

It was more that they would rather send Monarch an RPC than be connected to. Not everyone wants e.g. an HTTP server in their process. For example maybe they are security sensitive, or have a limited memory envelope, or other reasons.

And yeah, everyone used the same instrumentation library, which could be used in a lot of different contexts, sometimes surprising!

Again, a highly envious feature of a large organisation with almost exclusively bespoke applications that can port & integrate custom libraries directly into applications. Us little people have to contend with mostly black box applications, or occasionally native instrumented with at best a prom-alike endpoint.

Amusingly in the pre-web 1990's, at Telstra (Australia telco) we also developed & implemented a custom performance monitoring library that was integrated into in-house applications.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#106

Post by Too »

Jedd wrote: post #103
gttalbot wrote: post #92

Earlier quoted context omitted.

And yeah, everyone used the same instrumentation library, which could be used in a lot of different contexts, sometimes surprising!

Again, a highly envious feature of a large organisation with almost exclusively bespoke applications that can port & integrate custom libraries directly into applications. Us little people have to contend with mostly black box applications, or occasionally native instrumented with at best a prom-alike endpoint. Amusingly in the pre-web 1990's, at Telstra (Australia telco) we also developed & implemented a custom perf…

The rest of us can only hope Opentelemetry becomes more widely adopted. They have put in a lot of effort in decoupling the application instrumentation from the monitoring solution, to allow more rich instrumentation than just a prom-alike endpoint.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#107

Post by Too »

Thaxll wrote: post #98
gttalbot wrote: post #51

Earlier quoted context omitted.

Pull collection eventually became a real scaling bottleneck for Monarch. The way the "pull" collection worked was that there was an external process-discovery mechanism, which the leaf used to connect to the entities it was monitoring, the leaf backend processes would connect to the monitored entities to an endpoint that the collection library would listen on, and those entities collection libraries would stream the…

What issue originally did the pull model solved? Historically the push model existed before so what was the reason to move to a pull based solution?

https://prometheus.io/docs/introduction/faq/#why-do-you-pull... list a few reasons and also end with a note that it probably doesn't matter in the end. Personally, for smaller deployments, i like it because it gives you an easy overview of what should be running, otherwise you need to maintain this list elsewhere anyway, though today with all the auto-scaling around, the concept of "up" is getting more fuzzy.

On top of that there is also less risk that herd of misbehaving clients DoS the monitoring system, usually moments when you need such system the most. This of course wouldn't be a problem with a more scalable solution that distributes ingestion from querying, like the Monarch.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#108

Post by hintymad »

I don't quite get the benefit of pull model by default either. A pull model by default means that it's not easy for a library to publish its metrics. For instance, every god damn application is expected to implement a `/metrics` endpoint for a freaking agent to publish the application's metrics to Prometheus. With Monarch, any library or application can simply publish metrics to Monarch's API. Similarly in Netflix, publishing to its Altas system is totally transparent to library authors, with the help of their metric library.

Sometimes I feel many open source systems do not give a shit about productivity.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#110

Post by ignoramous »

gttalbot wrote: post #51
candiddevmike wrote: post #9

Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?

Pull collection eventually became a real scaling bottleneck for Monarch. The way the "pull" collection worked was that there was an external process-discovery mechanism, which the leaf used to connect to the entities it was monitoring, the leaf backend processes would connect to the monitored entities to an endpoint that the collection library would listen on, and those entities collection libraries would stream the…

Thanks.

What are some problems (or peculiarities that otherwise didn't exist) with the push based setup?

At another BigCloud, pull/push made for tasty design discussions as well, given the absurd scale of it all.

General consensus was, smaller fleet always pulls from its downstream; push only if downstream and upstream both have similar scaling characteristics.

Post reply on HN