Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

vldb.org

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#11

Post by joshuamorton »

fizixer wrote: post #7
kyrra wrote: post #3

Earlier quoted context omitted.

Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released. It tends to take a great amount of effort to open source anything at Google because of all the internal dependencies you have. It should be noted that Prometheus is an open source recreation of the precursor to monarch (developed b…

But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.

[Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it.

The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that isn't a large cloud provider need a custom solution like monarch. Prometheus or Datadog work fine for everyone else. This might be interesting reading for those companies, but also it might not be, because those can't be as centralized as monarch is (consider if prometheus had an API and ran a centralized cluster of data-ingestion servers, and you made time series to that global, Prometheus-owned, cluster).

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#12

Post by einpoklum »

fizixer wrote: post #7
kyrra wrote: post #3

Earlier quoted context omitted.

Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released. It tends to take a great amount of effort to open source anything at Google because of all the internal dependencies you have. It should be noted that Prometheus is an open source recreation of the precursor to monarch (developed b…

But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.

> aren't you giving away the core idea?

The core idea is a distributed time-series DBMS... not much to give away there. It "gives away" some architectural novelty, but it's not the solution to P=NP. These papers typically describe engineering feats more than they do a revoluntionary idea.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#13

Post by saddlerustle »

jeffbee wrote: post #10

Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…

Yeah the "supercomputer" ranking is a bit of a joke. Every mid-sized google dc would count as a top 10 supercomputer.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#14

Post by vii »

jeffbee wrote: post #10

Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…

they also estimate close to a petabyte of RAM; if all the RAM is spent in leaves that is about 6-7GB per leaf. I don't know that we can say it would be unreasonable to have 1 core per leaf replica; presumably some leaves have low utilisation so it might make sense to share the core with another workload. From a capacity planning standpoint, I think they leave this open though they do indicate that they are sometimes CPU bound so don't try to compress beyond delta compression. That might suggest multiple cores per leaf.

A single core design allows a very simple concurrency model, without having to worry about cache pingponging, false sharing, or myriad other issues. The parallelism is applied at higher layers, as there are multiple replicas for each leaf and obviously they can use many cores effectively overall.

I don't see that the paper gives enough information to help us prune the design space here.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#15

Post by jankeymeulen »

saddlerustle wrote: post #13
jeffbee wrote: post #10

Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…

Yeah the "supercomputer" ranking is a bit of a joke. Every mid-sized google dc would count as a top 10 supercomputer.

I work at one such "medium sized" Google DCs. Supercomputers are typically much more interconnected, whilst we have a much more traditional topology.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#16

Post by the-rc »

jeffbee wrote: post #10

Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…

2014: https://twitter.com/akpurtell/status/463747917782589441

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#17

Post by cortesoft »

fizixer wrote: post #7
kyrra wrote: post #3

Earlier quoted context omitted.

Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released. It tends to take a great amount of effort to open source anything at Google because of all the internal dependencies you have. It should be noted that Prometheus is an open source recreation of the precursor to monarch (developed b…

But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.

They aren’t saying it is difficult without giving away trade secrets, they are saying it is difficult because the software has dependencies on internal services, which themselves have dependencies on other internal services. Basically, in order to run it you need to also run the entirety of Google’s tech stack. This doesn’t work for an open source project.

They aren’t worried about ‘giving away the idea’, they just don’t have an easy technical way to open source just the one component.

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#19

Post by paxys »

jeffbee wrote: post #10

Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…

Is the entire internet the world's largest supercomputer?

Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

#20

Post by jldugger »

joshuamorton wrote: post #11
fizixer wrote: post #7

Earlier quoted context omitted.

But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.

[Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that…

The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful.

To the best of my knowledge, nothing else in the market does that.

Post reply on HN