[flagged]
Monarch: Google’s Planet-Scale In-Memory Time Series Database
61–70 of 133 posts
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#62Earlier quoted context omitted.
It seems like you're asserting a very specific definition of scalability that excludes Prometheus' scalability model. Scalability is an abstract property of a system that can be achieved in many different ways. It doesn't require any specific model of sharding, batching, query replication, etc. Do you not agree?
I’m not defining terms arbitrarily. https://en.wikipedia.org/wiki/Database_scalability Scalability means running a single workload across multiple machines. Prometheus intentionally does not scale this way. I’m not being mean, it is fact. It has made engineering design trade offs and one of those means it is not built to scale, this is fine, I’m not here pooping on your baby. You can build scalable systems on top of…
Scalability isn't a well-defined term, and Prometheus isn't a database. :shrug:
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#63Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#64Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?
Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
This. Any new large query or aggregation in the Borgmon/Prometheus model requires re-solving federation, and continuing maintenance of runtime configuration. That might technically be scalable in that you could do it but you have to maintain it, and pay the labor cost. It's not practical over a certain size or system complexity. It's also friction. You can only do the queries you can afford to set up.
That's why Google spent all that money to build Monarch. At the end of the day Monarch is vastly cheaper in person time and resources than manually-configured Borgmon/Prometheus. And there is much less friction in trying new queries, etc.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#65Earlier quoted context omitted.
It seems like you're asserting a very specific definition of scalability that excludes Prometheus' scalability model. Scalability is an abstract property of a system that can be achieved in many different ways. It doesn't require any specific model of sharding, batching, query replication, etc. Do you not agree?
Prometheus cannot evaluate a query over time series that do not fit in the memory of a single node, therefore it is not scalable. The fact that it could theoretically ingest an infinite amount of data that it cannot thereafter query is not very interesting.
It can? It just partitions the query over multiple nodes?
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#66Earlier quoted context omitted.
You pretty quickly exceed what one instance can handle for memory, cpu or both. At that point you don't have any real good options to scale while maintaining a flat namespace (you need to partition).
Sure? Prometheus scales with a federation model, not a single flat namespace.
This means each new query over a certain size becomes a federation problem, so the friction for trying new things becomes very high above the scale of a single instance.
Monitoring as a service has a lot of advantages.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#67Earlier quoted context omitted.
I’m not defining terms arbitrarily. https://en.wikipedia.org/wiki/Database_scalability Scalability means running a single workload across multiple machines. Prometheus intentionally does not scale this way. I’m not being mean, it is fact. It has made engineering design trade offs and one of those means it is not built to scale, this is fine, I’m not here pooping on your baby. You can build scalable systems on top of…
Scalability isn't a well-defined term, and Prometheus isn't a database. :shrug:
Wrong on both counts
Sorry for being rude, but this level of ignorance is extremely frustrating.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#68Earlier quoted context omitted.
Sure? Prometheus scales with a federation model, not a single flat namespace.
This means each new query over a certain size becomes a federation problem, so the friction for trying new things becomes very high above the scale of a single instance. Monitoring as a service has a lot of advantages.
Well you obviously don't issue metrics queries over arbitrarily large datasets, right? The Prometheus architecture reflects this invariant. You constrain queries against both time and domain boundaries.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#69Earlier quoted context omitted.
Is it really that different from, say, the way Prometheus supports histogram-based quantiles? https://prometheus.io/docs/practices/histograms/ Granted, it looks like Monarch supports a more cleanly-defined schema for distributions, whereas Prometheus just relies on you to define the buckets yourself and follow the convention of using a "le" label to expose them. But the underlying representation (an empirical CDF) se…
Much different. When you are reporting histograms you can combine them and see the true p50 or whatever across all the individual systems reporting the metric.
Can you elaborate a bit? You can do the same in Prometheus by summing the bucket counts. Not sure what you mean by “true p50” either. With buckets it’s always an approximation based on the bucket widths.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#70Earlier quoted context omitted.
This means each new query over a certain size becomes a federation problem, so the friction for trying new things becomes very high above the scale of a single instance. Monitoring as a service has a lot of advantages.
Well you obviously don't issue metrics queries over arbitrarily large datasets, right? The Prometheus architecture reflects this invariant. You constrain queries against both time and domain boundaries.
Monarch can support both ad-hoc and periodic, standing queries of arbitrarily large size, and has the means to spread the computation out over many intermediate mixer and leaf nodes. It does query push-down so that the "expensive" parts of aggregations, joins, etc., can be done in massively parallel fashion at the leaf level.
It scales so well that many aggregations are set up and computed for every service across the whole company (CPU, memory usage, error rates, etc.). For basic monitoring you can run a new service in production and go and look at a basic dashboard for it without doing anything else to set up monitoring.