Given the size of the beast, I wonder what monitors it?
Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
41–49 of 49 posts
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
#42Given the size of the beast, I wonder what monitors it?
A borgmon
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
#43Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…
We don't run exacycle any more (I built and ran exacycle for several years). It's not a cost-effective way to do science, but yes, the scale was absolutely insane. These days, I'm more interesting in seeing if there are ways to use TPUs, rather than CPUs, for similar kinds of opportunistic computing.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
#44Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…
Yeah the "supercomputer" ranking is a bit of a joke. Every mid-sized google dc would count as a top 10 supercomputer.
supercomputers, by tradition, are tightly coupled in a way that Google datacenter servers aren't. The closest thing Google has to supercomputers are GPUs linked by high performance networks, and TPUs (which have their own custom toroidal mesh).
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
#45Earlier quoted context omitted.
[Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that…
The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful. To the best of my knowledge, nothing else in the market does that.
Circonus (https://www.circonus.com/) supports both recording histogram directly, and merging histograms for analysis. IIUC, it also supports first-class timeseries data similarly to Monarch, where each data point has a high precision timestamp that does not have to align with other timeseries in the data set.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
#46Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…
It's not a supercomputer.
The reason supercomputers are so tiny compared to cloud data-centers is that cloud computing is highly and mostly-triviallly parallel, but supercomputers are mesh and serial.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
#47Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…
It's not a supercomputer. The reason supercomputers are so tiny compared to cloud data-centers is that cloud computing is highly and mostly-triviallly parallel, but supercomputers are mesh and serial.
That appears to be your own private definition of the term. There are lots of things in the top500 list that are just a pile of Xeon boxes with Ethernet.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
#48Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…
When I had done the presentation at Facebook's @Scale NYC conference last summer, I talked the powers that be into using more specific numbers, and they allowed it for the paper too.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]
#49Earlier quoted context omitted.
The query language is the brainchild of John Banning, one of the authors of the paper, and has a long history behind it. In 2007 or so he started working on a replacement for Borgmon's rule language; the thinking at the time was that the main problem with Borgmon was that its language was surprising and difficult for casual users to grasp. (And with a monitoring language, there are only casual users.) That work event…
Googlers still have to use the terrible python dsl (“mash”). Even worse: they have to use it wrapped in a different terrible python dsl (“gmon”). Sigh.
I was mostly using th new notation when I left in 2018.