21–30 of 49 posts
Question for googlers (and ex-googlers). Is there anything out there that's as convenient as /streamz, but in opensource form?
If you really want /streamz with simple aggregation for distribution, description for metrics etc in an HTML page, I haven't encountered one yet.
Earlier quoted context omitted.
[Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that…
The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful. To the best of my knowledge, nothing else in the market does that.
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/Typed...
Since the distribution is represented by a CDF of buckets, there's no guarantee that you'll get an accurate representation of the median or any other quantile. On the other hand you'll get an exact average.
Question for googlers (and ex-googlers). Is there anything out there that's as convenient as /streamz, but in opensource form?
If you are looking for /varz, https://prometheus.io If you really want /streamz with simple aggregation for distribution, description for metrics etc in an HTML page, I haven't encountered one yet.
Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-yea…
Yeah the "supercomputer" ranking is a bit of a joke. Every mid-sized google dc would count as a top 10 supercomputer.
Question for googlers (and ex-googlers). Is there anything out there that's as convenient as /streamz, but in opensource form?
Question for googlers (and ex-googlers). Is there anything out there that's as convenient as /streamz, but in opensource form?
The most interesting idea is that its native data types are time series of integers, doubles, distributions of doubles, booleans, and tuples of the above. This means that the data you operate on intrinsically consist of many timestamped data points. It's easy to apply an operation to each point of the data, and it's also easy to apply operations on a rolling window, or on successive points. This makes the language have the feel of an array-based language, but even better because the elements are timestamped and the array can be sparse.
Furthermore the presence of fields in each data point adds more dimensions to the level of aggregation (not just the inherent time-based). Now the language has the feel of a native multi-dimensional array language. It feels amazing to program in it. You can easily do sophisticated queries like figuring out how many standard deviations each task's RPC latency for a specific call is above or below all tasks' mean, for outlier detection.
As someone interested in PL theory, I've long found the query language exposed by Monarch surprisingly interesting (briefly discussed in section 5.1 but the description doesn't quite do it justice). It's a functional language, a breath of fresh air compared to "real programming languages" in use at Google like C++, Java, or Go. The most interesting idea is that its native data types are time series of integers, doubl…
As someone interested in PL theory, I've long found the query language exposed by Monarch surprisingly interesting (briefly discussed in section 5.1 but the description doesn't quite do it justice). It's a functional language, a breath of fresh air compared to "real programming languages" in use at Google like C++, Java, or Go. The most interesting idea is that its native data types are time series of integers, doubl…
If you ignore the issue of alignment, I actually think that a more conventional array based language, either something SQL-like or numpy-like, would be more accessible to most people. And things like windowing are incredibly unintuitive for most users.
Earlier quoted context omitted.
But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.
[Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that…
I'm not actually sure that's true. It's like many other things inside Google - people outside don't necessarily understand the value or know what's actually possible, because they've never experienced anything similar. It's sort of like trying to discuss the finer points of the taste of oysters with someone who has never tasted them.
I would very much like the feature set of Monarch (and streamz), without the maintenance overhead or even insane scale. Very, very few companies out there need to run anything at anywhere near "billion-user" scale, but literally all of them could benefit from painless and detailed monitoring that Monarch offers.