Discussion from 2020: https://news.ycombinator.com/item?id=24303422
Monarch: Google’s Planet-Scale In-Memory Time Series Database
11–20 of 133 posts
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#12A lot of Google projects seem to rely on other Google projects. In this case Monarch relies on spanner. I guess its nice to publish at least the conceptual design so that others can implement it in “rest of the world” case. Working with OSS can be painful, slow and time consuming so this seems like a reasonable middle ground (although selfishly I do wish all of this was source available).
I don't think there's any spanner necessity and iirc monarch existed pre-spanner.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#13Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?
Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
That's what Mimir solves
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#14Google Cloud Monitoring's time series database is backed by Monarch.
The query language is mql which closely resembles the internal Python based query language: https://cloud.google.com/monitoring/mql
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#15A lot of Google projects seem to rely on other Google projects. In this case Monarch relies on spanner. I guess its nice to publish at least the conceptual design so that others can implement it in “rest of the world” case. Working with OSS can be painful, slow and time consuming so this seems like a reasonable middle ground (although selfishly I do wish all of this was source available).
Spanner may be hard to set up even with source code available. It relies on atomic clocks for reliable ordering of events.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#16Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?
It's sort of a pull/push hybrid. The client connects to the collection system and is told how often to send each metric (or group of them) back over that same connection. You configure per target/metric collection policy centrally.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#17Interesting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?
Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
Can you elaborate? I’ve ran Prometheus at some scale and it’s performed fine.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#18Google Cloud Monitoring's time series database is backed by Monarch. The query language is mql which closely resembles the internal Python based query language: https://cloud.google.com/monitoring/mql
MQL is an improvement over the internal language, IMO. There are some missing features around literal tables, but otherwise the language is more consistent and flexible.
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#19I broke this once several years ago. I even use the incident number in my random usernames to see if a Googler recognizes it.
How did you break it?
Re: Monarch: Google’s Planet-Scale In-Memory Time Series Database
#20Earlier quoted context omitted.
Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
Can you elaborate? I’ve ran Prometheus at some scale and it’s performed fine.
You pretty quickly exceed what one instance can handle for memory, cpu or both. At that point you don't have any real good options to scale while maintaining a flat namespace (you need to partition).