Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monarch: Google’s Planet-Scale In-Memory Time Series Database

micahlerner.com

Post reply on HN
Print view View original

91–100 of 133 posts

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#91

Post by gttalbot »

Jedd wrote: post #89
gttalbot wrote: post #51

Earlier quoted context omitted.

Pull collection eventually became a real scaling bottleneck for Monarch. The way the "pull" collection worked was that there was an external process-discovery mechanism, which the leaf used to connect to the entities it was monitoring, the leaf backend processes would connect to the monitored entities to an endpoint that the collection library would listen on, and those entities collection libraries would stream the…

Thanks George, and apologies for missing this comment on my first scan through this page. Your Youtube talk is lined up for viewing later today. We're using prom + cortex/mimir. With ~30-60k hosts + at least that figure again for other endpoints (k8s, snmp, etc), so we can get away with semi-manual sharding (os, geo, env, etc). We're happy with 1m polling, which is still maybe 50 packets per query, but no persistent…

It was more that they would rather send Monarch an RPC than be connected to. Not everyone wants e.g. an HTTP server in their process. For example maybe they are security sensitive, or have a limited memory envelope, or other reasons.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#92

Post by gttalbot »

gttalbot wrote: post #91
Jedd wrote: post #89

Earlier quoted context omitted.

Thanks George, and apologies for missing this comment on my first scan through this page. Your Youtube talk is lined up for viewing later today. We're using prom + cortex/mimir. With ~30-60k hosts + at least that figure again for other endpoints (k8s, snmp, etc), so we can get away with semi-manual sharding (os, geo, env, etc). We're happy with 1m polling, which is still maybe 50 packets per query, but no persistent…

It was more that they would rather send Monarch an RPC than be connected to. Not everyone wants e.g. an HTTP server in their process. For example maybe they are security sensitive, or have a limited memory envelope, or other reasons.

And yeah, everyone used the same instrumentation library, which could be used in a lot of different contexts, sometimes surprising!

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#93

Post by gttalbot »

kasey_junk wrote: post #47
hn_go_brrrrr wrote: post #28

Earlier quoted context omitted.

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…

My personal opinion is that they should have done a log linear histogram which solves the problems you mention (with other trade offs) but to me the big news was making the db flexible enough to have that data type. Leaving the world of single numeric type for each datum will influence the next generation of open source metrics db.

Yeah in theory people could do their own custom bucketing functions. Would be worth researching log-linear for that certainly.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#94

Post by gttalbot »

sydthrowaway wrote: post #32

Stop overhyping software with buzzwords

What? Planet scale? Well. You can literally issue a query that fans out to every continent on Earth, and returns the result right to your dashboard. Not exaggerating. ;^) (OK maybe not Antarctica but I'm not sure...)

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#95

Post by wbl »

praptak wrote: post #15
pm90 wrote: post #5

A lot of Google projects seem to rely on other Google projects. In this case Monarch relies on spanner. I guess its nice to publish at least the conceptual design so that others can implement it in “rest of the world” case. Working with OSS can be painful, slow and time consuming so this seems like a reasonable middle ground (although selfishly I do wish all of this was source available).

Spanner may be hard to set up even with source code available. It relies on atomic clocks for reliable ordering of events.

Atomic clocks aren't that exotic, and a GPS disciplined ovenized quartz oscillator will do just fine outside of a disruption. The hard part is getting the right sampling semantics, requiring end to end error analysis.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#96

Post by wbl »

preseinger wrote: post #49
dijit wrote: post #46

Earlier quoted context omitted.

It is not. It basically does the opposite of what every scalable system does. To get HA you double you’re number of pollers. To get scale your queries you aggregate them into other prometheii. If this is scalability: everything is scalable.

I don't understand how the properties you're describing imply that Prometheus isn't scalable. High Availability always requires duplication of effort. Scaling queries always requires sharding and aggregation at some level. I've deployed stock Prometheus at global scale, O(100k) targets, with great success. You have to understand and buy into Prometheus' architectural model, of course.

And I've seen a system that did that. We had to predict what global aggregates we would need. Looking at a metric required finding the right instance to connect to if it wasn't in the bubbleup. Picking the right expressions to avoid double counts was hard. Want to do something fancy? No luck because of the lack of distributed querying.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#97

Post by xyzzy_plugh »

preseinger wrote: post #68
gttalbot wrote: post #66

Earlier quoted context omitted.

This means each new query over a certain size becomes a federation problem, so the friction for trying new things becomes very high above the scale of a single instance. Monitoring as a service has a lot of advantages.

Well you obviously don't issue metrics queries over arbitrarily large datasets, right? The Prometheus architecture reflects this invariant. You constrain queries against both time and domain boundaries.

That's a choice Prometheus has made, not an invariant. Many scalable systems support arbitrarily large queries as they themselves scale. Prometheus pretty much prevents creating arbitrarily large datasets to begin with, so the point is kind of moot.

Requiring query authors to understand the arrangement of their aggregation layer seems like a reasonable idea but is in fact quite ridiculous.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#98

Post by Thaxll »

gttalbot wrote: post #51
candiddevmike wrote: post #9

Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?

Pull collection eventually became a real scaling bottleneck for Monarch. The way the "pull" collection worked was that there was an external process-discovery mechanism, which the leaf used to connect to the entities it was monitoring, the leaf backend processes would connect to the monitored entities to an endpoint that the collection library would listen on, and those entities collection libraries would stream the…

What issue originally did the pull model solved? Historically the push model existed before so what was the reason to move to a pull based solution?

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#99

Post by foota »

preseinger wrote: post #82
dijit wrote: post #76

Earlier quoted context omitted.

It is extremely unbecoming to lie about who you are on this forum. Scalability is defined differently depending on context; in this context (a monitoring/time series solution) it is defined as being able to hold a dataset larger than a single machine that scales horizontally. Downsampling the data or transforming it does not meet that criteria, since that’s no longer the original data. The way Prometheus “scales” tod…

> in this context (a monitoring/time series solution) it is defined as being able to hold a dataset larger than a single machine that scales horizontally. This just isn't true :shrug: Horizontal scaling is one of many strategies.

I think the disconnect is that promethus helps a user to shard things, but it's not automatic. Other time series databases and monitoring solutions automatically distribute and query across servers. It's like postgres vs newswl (aka foundationdb, spanner, etc.,).

While Prometheus supports sharding queries when a user sets it up, my understanding is that this has to be done manually, which is definitely less convenient. This is better than a hypothetical system that doesn't allow this at all, but still not the same as something that handles scaling magically.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#100

Post by NHQ »

kasey_junk wrote: post #25

A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.

Is this lossy?
Post reply on HN