Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monarch: Google’s Planet-Scale In-Memory Time Series Database

micahlerner.com

Post reply on HN
Print view View original

81–90 of 133 posts

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#81

Post by buro9 »

kasey_junk wrote: post #25

A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.

Prometheus is adding sparse histograms. There's a couple of online talks about it already but one of the maintainers, Ganesh, is giving a talk at kubecon on it next week off anyone is attending and curious about it.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#82

Post by preseinger »

dijit wrote: post #76
preseinger wrote: post #73

Earlier quoted context omitted.

Ignorance? I'm a core Prometheus contributor and a 20+ year distsys veteran. Prometheus is not a database, and scalability is not well-defined. These are not controversial statements.

It is extremely unbecoming to lie about who you are on this forum. Scalability is defined differently depending on context; in this context (a monitoring/time series solution) it is defined as being able to hold a dataset larger than a single machine that scales horizontally. Downsampling the data or transforming it does not meet that criteria, since that’s no longer the original data. The way Prometheus “scales” tod…

> in this context (a monitoring/time series solution) it is defined as being able to hold a dataset larger than a single machine that scales horizontally.

This just isn't true :shrug: Horizontal scaling is one of many strategies.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#83

Post by gttalbot »

Jedd wrote: post #78
lokar wrote: post #16

Earlier quoted context omitted.

It's sort of a pull/push hybrid. The client connects to the collection system and is told how often to send each metric (or group of them) back over that same connection. You configure per target/metric collection policy centrally.

So, much like a Zabbix agent, with both active (push) & passive (pull) capabilities. We're diving into OTEL, and the registration / discovery challenges don't seem to have any kind of best-practice consensus out there. We're looking at NodeRED (telegraf agent can query from same at startup) but it brings its own challenges. I haven't read the full paper, but do you know if the push model was revisited mostly for auto…

See my comment below, on the challenges of pull based collection on Monarch. There were many. I can answer questions, if that's helpful.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#84

Post by gttalbot »

Jedd wrote: post #78
lokar wrote: post #16

Earlier quoted context omitted.

It's sort of a pull/push hybrid. The client connects to the collection system and is told how often to send each metric (or group of them) back over that same connection. You configure per target/metric collection policy centrally.

So, much like a Zabbix agent, with both active (push) & passive (pull) capabilities. We're diving into OTEL, and the registration / discovery challenges don't seem to have any kind of best-practice consensus out there. We're looking at NodeRED (telegraf agent can query from same at startup) but it brings its own challenges. I haven't read the full paper, but do you know if the push model was revisited mostly for auto…

Also, I gave this talk a couple of years ago, though I'm not sure how deeply I talked about collection models.

https://youtu.be/2mw12B7W7RI

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#85

Post by mherdeg »

heinrichhartman wrote: post #45
hn_go_brrrrr wrote: post #28

Earlier quoted context omitted.

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…

Circonus Histograms solve that by using a universal bucketing scheme. Details are explained in this paper: https://arxiv.org/abs/2001.06561 Disclaimer: I am a co-author.

Wow, this is a fantastic solution to some questions I've had rattling around in my head for years about the optimal bucket choices to minimize error given a particular set of buckets.

Do I read right that circllhist has a pretty big number of bin sizes and is not configurable (except that they're sparse so may be small on disk)?

I've found myself using high-cardinality Prometheus metrics where I can only afford 10-15 distinct histogram buckets. So I end up

(1) plugging in my live system data from normal operations and from outage periods into various numeric algorithms that propose optimal bucket boundaries. These algorithms tell me that I could get great accuracy if I chose thousands of buckets, which, thanks for rubbing it in about my space problems :(. Then I write some more code to collapse those into 15 buckets while minimizing error at various places (like p50, p95, p99, p999 under normal operations and under irregular operations).

(2) making sure I have an explicit bucket boundary at any target that represents a business objective (if my service promises no more than 1% of requests will take >2500ms, setting a bucket boundary at 2500ms gives me perfectly precise info about whether p99 falls above/below 2500ms)

(3) forgetting to tune this and leaving a bunch of bad defaults in place which often lead to people saying "well, our graph shows a big spike up to 10000ms but that's just because we forgot to tune our histogram bucket boundaries before the outage, actually we have to refer to logs to see the timeouts at 50 sec"

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#86

Post by gttalbot »

klysm wrote: post #4

I don’t really grasp why this is a useful spot in the trade off space from a quick skim. Seems risky.

The key trade off if low-dependency, with everything up to and including alert notification delivery pushed down to the region. For alerting queries to happen there is little infrastructure Monarch depends upon to keep running beyond networking and being scheduled on the machines.

If you think about Bigtable, a key observation that the Monarch team made very early on is that, if you can support good materialized views (implemented as periodic standing queries) written back to the memtable, and the memtable can hold the whole data set needed to drive alerting, this can work even if much of Google's infrastructure is having problems. It also allows Monarch to monitor systems like Bigtable, Colossus, etc., as it doesn't use them as serving dependencies for alerting or recent dashboard data.

It's a question of optimizing for graceful degradation in the presence of failure of the infrastructure around the monitoring system. The times the system will experience its heaviest and most unpredictable load will be when everyone's trying to figure out why their service isn't working.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#87

Post by gttalbot »

hn_go_brrrrr wrote: post #28
kasey_junk wrote: post #25

A huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be…

Yeah it was a tough tradeoff for the default case, because the team didn't want to use too much memory in everyone's binary since the RPC metrics were on by default. This is easily changeable by the user if necessary, though.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#88

Post by bbkane »

dijit wrote: post #8
klysm wrote: post #4

I don’t really grasp why this is a useful spot in the trade off space from a quick skim. Seems risky.

There’s a good talk on Monarch https://youtu.be/2mw12B7W7RI Why it exists is laid out quite plainly. The pain of it is we’re all jumping on Prometheus (borgmon) without considering why Monarch exists. Monarch doesn’t have a good corollary outside of google. Maybe some weird mix of timescale DB backed by cockroachdb with a Prometheus push gateway.

They should open source it like they did Kubernetes. Otherwise the world will (continue to) converge on the Prometheus model and Google will be left with this weird system that may be technically better but is unfamiliar to incoming engineers

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#89

Post by Jedd »

gttalbot wrote: post #51
candiddevmike wrote: post #9

Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?

Pull collection eventually became a real scaling bottleneck for Monarch. The way the "pull" collection worked was that there was an external process-discovery mechanism, which the leaf used to connect to the entities it was monitoring, the leaf backend processes would connect to the monitored entities to an endpoint that the collection library would listen on, and those entities collection libraries would stream the…

Thanks George, and apologies for missing this comment on my first scan through this page. Your Youtube talk is lined up for viewing later today.

We're using prom + cortex/mimir. With ~30-60k hosts + at least that figure again for other endpoints (k8s, snmp, etc), so we can get away with semi-manual sharding (os, geo, env, etc). We're happy with 1m polling, which is still maybe 50 packets per query, but no persistent conns held open to agents.

I'm guessing your TCP issues were exacerbated by a much high polling frequency requirement? You come back to persistent connections a lot, so this sounds like a bespoke agent, and/or the problem was not (mostly) a connection establish/tear-down performance issue?

The external discovery service - I assume an in-house, and now long disappeared and not well publicly described system? ;) We're looking at NodeRED to fill that gap, so it also becomes a critical component, but the absence only bites at agent restart. We're pondering wrapping some code around the agents to be smarter about dealing with a non-responsive config service. (During a major incident we have to assume a lot of things will be absent and/or restarting.)

The concerns around incoming conns to their apps, it sounds like those same teams you were dealing with ended up having to instrument their code with something from you anyway -- was it the DoS risk they were concerned about?

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database

#90

Post by gttalbot »

lokar wrote: post #79
Jedd wrote: post #78

Earlier quoted context omitted.

So, much like a Zabbix agent, with both active (push) & passive (pull) capabilities. We're diving into OTEL, and the registration / discovery challenges don't seem to have any kind of best-practice consensus out there. We're looking at NodeRED (telegraf agent can query from same at startup) but it brings its own challenges. I haven't read the full paper, but do you know if the push model was revisited mostly for auto…

Push can help a bit, but you still have to know which endpoints you expect to hear from (if you want to detect they are missing).

If you can put these into the metrics system as metrics themselves, then join against them, that works. (IIRC handling all the various things that could be happening in a data center like drains, planned & unplanned downtime, etc., it's more complex than a simple join.)
Post reply on HN