Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prometheus: An open-source service monitoring system and time series database

prometheus.io

Post reply on HN
Print view View original

71–80 of 122 posts

Re: Prometheus: An open-source service monitoring system and time series database

#71

Post by falcolas »

SEJeff wrote: post #64

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solu…

Before digging into converting the backend to Prometheus, you may want to have a look at their actual persistence method. What worries me is that they are still storing one metric per file, and are copying data from one file to the next to perform checkpoints or purges (which is admittedly an improvement over whisper's method of thrashing back and forth across a single file).

They have the code to do in-place method of appending chunks, but in my admittedly brief foray through the code I could not actually find any use of it.

Re: Prometheus: An open-source service monitoring system and time series database

#73

Post by SEJeff »

falcolas wrote: post #71
SEJeff wrote: post #64

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solu…

Before digging into converting the backend to Prometheus, you may want to have a look at their actual persistence method. What worries me is that they are still storing one metric per file, and are copying data from one file to the next to perform checkpoints or purges (which is admittedly an improvement over whisper's method of thrashing back and forth across a single file). They have the code to do in-place method…

The graphite backend is pluggable. It isn't converting so much as adding support for. One of my favorite backends is Cyanite which ultimately stores the data in Cassandra.

Re: Prometheus: An open-source service monitoring system and time series database

#74

Post by bbrazil »

bhuga wrote: post #69
jrv wrote: post #63

Earlier quoted context omitted.

Brian already hinted at this, but the main reason why we do this is to have coherency of the data model across the whole chain: client app -> Prometheus server -> alerting rules -> alert handling This allows e.g. initial dimensional labels from the clients to cleanly propagate all the way into alerts, and allows you to silence, aggregate, or route alerts by any label combination. If OpenTSDB-style dimensional metrics…

I haven't used Prometheus yet, but I'm not sure I agree that it makes sense to send dimensional time series all the way down the alert chain. It allows full-featured alerts to be used, yes. But you're not going to be able to put all of your alert logic into the prometheus chain, so you're going to end up having alert rules in at least two places. For example, in response to the latest security hullabaloo, we recently…

> This isn't a time series check; it's just an alert condition.

I think you're conflating instrumentation and alerting.

Many systems only offer alert conditions on very low-level data that's tightly-bound to the instrumentation (e.g. a single check on a single host), Prometheus is more powerful as the instrumentation and alerting are separate, with dimensions adding further power.

Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom).

You can then setup an alert for when the value is greater than 0 on any host (ALERT OldLibcAlert IF old_libc_process > 0 WITH ...). The advantage of having all the dimensions is that you can analyse that over time, and graph how many hosts had a problem to trend how well you're dealing with the libc issue.

A big win for having dimensions for alerting all the way up the stack is in silencing, notification and throttling of related alerts. You can slice and dice and have things as granular as makes sense for your use case.

> Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline.

There is an alert bridge to nagios if you want to send your alerts out that way. You'll lose some of the rich data that the dimensions provide though.

Re: Prometheus: An open-source service monitoring system and time series database

#75

Post by jrv »

falcolas wrote: post #71
SEJeff wrote: post #64

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solu…

Before digging into converting the backend to Prometheus, you may want to have a look at their actual persistence method. What worries me is that they are still storing one metric per file, and are copying data from one file to the next to perform checkpoints or purges (which is admittedly an improvement over whisper's method of thrashing back and forth across a single file). They have the code to do in-place method…

Which storage/purge strategy would you recommend? (see also this thread for more discussion about the storage: https://news.ycombinator.com/item?id=8996832)

Re: Prometheus: An open-source service monitoring system and time series database

#76

Post by falcolas »

jrv wrote: post #70
falcolas wrote: post #68

Earlier quoted context omitted.

So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code…

So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten…

> does it work well for time series data

It's a key/value store at its heart, with all the ACID magic and memory buffering built in.

Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates).

Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data.

Re: Prometheus: An open-source service monitoring system and time series database

#77

Post by chris_va »

ggambetta wrote: post #2

"Those who cannot remember the Borgmon are doomed to repeat it" ;) Just kidding, this is looking really good, I hope to get some hands-on experience with it soon.

After seeing the Prometheus query language, I opened the comments looking for this quote :).

Re: Prometheus: An open-source service monitoring system and time series database

#78

Post by beorn7 »

falcolas wrote: post #68
beorn7 wrote: post #61

Earlier quoted context omitted.

We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.

So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code…

The checkpointing is just a way to minimize sample losses from unfinished chunks upon a crash. If you don't care or if you don't crash, you can disable checkpointing. Normal operation will be the same.

Invalidation of old data is super easy with the chunked files as you can simply "truncate from the beginning", which is supported by various file systems (e.g. XFS). However, benchmarks so far showed that the relative amount of IO for purging old chunks is so small compared to overall IO that we didn't bother to implement it yet. Could be done if it turns out to be critical.

Re: Prometheus: An open-source service monitoring system and time series database

#79

Post by jrv »

falcolas wrote: post #76
jrv wrote: post #70

Earlier quoted context omitted.

So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten…

> does it work well for time series data It's a key/value store at its heart, with all the ACID magic and memory buffering built in. Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates). Issuing updates instead of deletes is possible because you know the storage durat…

An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout:

[time series fingerprint : time range] -> [chunk of ts/value samples]]

At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the contents the value), except that they also allow efficient appends to values.

> Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data.

Do you mean you would actually append/update an existing value in the KV-store (which most KV stores don't allow without reading/writing the whole key)?

Re: Prometheus: An open-source service monitoring system and time series database

#80

Post by simple10 »

Looks really promising for smaller clusters. However, the pull/scraping model for stats could be problematic for larger scale.

I've been experimenting with metrics collection using heka (node) -> amqp -> heka (aggregator) -> influxdb -> grafana. It works extremely well and scales nicely but requires writing lua code for anomaly detection and alerts – good or bad depending on your preference.

I highly recommend considering Heka[1] for shipping logs to both ElasticSearch and InfluxDB if you need more scale and flexibility than Prometheus currently provides.

[1] https://github.com/mozilla-services/heka

Post reply on HN