Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prometheus: An open-source service monitoring system and time series database

prometheus.io

Post reply on HN
Print view View original

81–90 of 122 posts

Re: Prometheus: An open-source service monitoring system and time series database

#81

Post by thesnider »

ggambetta wrote: post #2

"Those who cannot remember the Borgmon are doomed to repeat it" ;) Just kidding, this is looking really good, I hope to get some hands-on experience with it soon.

After working at a job that had a horrible patchwork of monitoring techniques (including at least two in-house systems), I was desperately pining for borgmon, actually. Never thought those words would come out of my mouth.

This does seem to have addressed at least a couple of the issues with that system, in that its config language is sane, and its scrape format is well-defined and typed.

Re: Prometheus: An open-source service monitoring system and time series database

#82

Post by bbrazil »

simple10 wrote: post #80

Looks really promising for smaller clusters. However, the pull/scraping model for stats could be problematic for larger scale. I've been experimenting with metrics collection using heka (node) -> amqp -> heka (aggregator) -> influxdb -> grafana. It works extremely well and scales nicely but requires writing lua code for anomaly detection and alerts – good or bad depending on your preference. I highly recommend consid…

> However, the pull/scraping model for stats could be problematic for larger scale.

From experience of similar systems at massive scale, I expect no scaling problems with pulling in and of itself. Indeed, there's some tactical operational options you get with pull that you don't have with push. See http://www.boxever.com/push-vs-pull-for-monitoring for my general thoughts on the issue.

> InfluxDB

InfluxDB seems best suited for event logging rather than systems monitoring. See also http://prometheus.io/docs/introduction/comparison/#prometheu...

Re: Prometheus: An open-source service monitoring system and time series database

#83

Post by KyleBrandt »

bhuga wrote: post #55

It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted for monitoring, notifications or graphing as nginx is for a web server. Not that I'd do a better job, but every time I further configure our monitoring system, I get that feeling that we're missing something as an industry. It's a space with lots of tools tha…

Shameless plug. But Bosun's focus is largely on Alerting (http://bosun.org). We have an expression language built in that allows for complex rules. It leverages OpenTSDBs multi dimensional facet to make alert instantiations, but you can also change the scope by transposing your results.

It also now supports Logstash and Graphite as backends as well. The Graphite support is thanks to work at Vimeo.

Another nice thing about Bosun is you can test your alerts against time series history to see when they would have triggered so you can largely tune them before you commit them to production.

Re: Prometheus: An open-source service monitoring system and time series database

#84

Post by bhuga »

bbrazil wrote: post #74
bhuga wrote: post #69

Earlier quoted context omitted.

I haven't used Prometheus yet, but I'm not sure I agree that it makes sense to send dimensional time series all the way down the alert chain. It allows full-featured alerts to be used, yes. But you're not going to be able to put all of your alert logic into the prometheus chain, so you're going to end up having alert rules in at least two places. For example, in response to the latest security hullabaloo, we recently…

> This isn't a time series check; it's just an alert condition. I think you're conflating instrumentation and alerting. Many systems only offer alert conditions on very low-level data that's tightly-bound to the instrumentation (e.g. a single check on a single host), Prometheus is more powerful as the instrumentation and alerting are separate, with dimensions adding further power. Prometheus has many instrumentation…

> I think you're conflating instrumentation and alerting.

A bit, because I'm hesitant to cram into a time series database data that I consider boolean. I'd move past this for consistency if the checks were easy to create, though.

> Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom).

Interesting. That architecture or configuration of this exporter isn't documented yet, but if just magically sent numbers upstream from text files that I keep up to date, that might be worth a migration by itself. Nagios SSH checks are ridiculous to manage.

Thanks for indulging my conversation in any case. I'll put this on my radar and watch where it goes.

Re: Prometheus: An open-source service monitoring system and time series database

#85

Post by jrv »

KyleBrandt wrote: post #83
bhuga wrote: post #55

It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted for monitoring, notifications or graphing as nginx is for a web server. Not that I'd do a better job, but every time I further configure our monitoring system, I get that feeling that we're missing something as an industry. It's a space with lots of tools tha…

Shameless plug. But Bosun's focus is largely on Alerting ( http://bosun.org ). We have an expression language built in that allows for complex rules. It leverages OpenTSDBs multi dimensional facet to make alert instantiations, but you can also change the scope by transposing your results. It also now supports Logstash and Graphite as backends as well. The Graphite support is thanks to work at Vimeo. Another nice thin…

Great. Yeah, adding unit tests for alerts is something that we still need to add the capability for. But at least you can always already manually graph your alert expressions and see how they would have evaluated over time.

Interesting work on Bosun by the way! Seems like there is quite some overlap with Prometheus, but I yet have to study it in depth. Is my impression correct that OpenTSDB is a requirement, or is there any local storage component? I guess you could run OpenTSDB colocated on a single node...

Re: Prometheus: An open-source service monitoring system and time series database

#86

Post by simple10 »

bbrazil wrote: post #82
simple10 wrote: post #80

Looks really promising for smaller clusters. However, the pull/scraping model for stats could be problematic for larger scale. I've been experimenting with metrics collection using heka (node) -> amqp -> heka (aggregator) -> influxdb -> grafana. It works extremely well and scales nicely but requires writing lua code for anomaly detection and alerts – good or bad depending on your preference. I highly recommend consid…

> However, the pull/scraping model for stats could be problematic for larger scale. From experience of similar systems at massive scale, I expect no scaling problems with pulling in and of itself. Indeed, there's some tactical operational options you get with pull that you don't have with push. See http://www.boxever.com/push-vs-pull-for-monitoring for my general thoughts on the issue. > InfluxDB InfluxDB seems best…

Good point on push-vs-pull. I'm biased towards push because of microservices that behave like batch jobs. In effect, I'm using AMQP in a similar way as the Prometheus pushgateway.

Agreed that InfluxDB is suited for event logging out of the box, but the March 2014 comparison of Influx is outdated IMO.

I'm using Heka to send numeric time series data to Influx and full logs to ElasticSearch. It's possible to send full logs to non-clustered Influx in 0.8, but it's useful to split out concerns to different backends.

I also like that Influx 0.9 dropped LevelDB support for BoltDB. There will be more opportunity for performance enhancements.

Re: Prometheus: An open-source service monitoring system and time series database

#87

Post by ithkuil »

jrv wrote: post #79
falcolas wrote: post #76

Earlier quoted context omitted.

> does it work well for time series data It's a key/value store at its heart, with all the ACID magic and memory buffering built in. Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates). Issuing updates instead of deletes is possible because you know the storage durat…

An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout: [time series fingerprint : time range] -> [chunk of ts/value samples]] At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the content…

Did your previous leveldb approach perform way worse on reads, writes or both?

Re: Prometheus: An open-source service monitoring system and time series database

#88

Post by bbrazil »

bhuga wrote: post #84
bbrazil wrote: post #74

Earlier quoted context omitted.

> This isn't a time series check; it's just an alert condition. I think you're conflating instrumentation and alerting. Many systems only offer alert conditions on very low-level data that's tightly-bound to the instrumentation (e.g. a single check on a single host), Prometheus is more powerful as the instrumentation and alerting are separate, with dimensions adding further power. Prometheus has many instrumentation…

> I think you're conflating instrumentation and alerting. A bit, because I'm hesitant to cram into a time series database data that I consider boolean. I'd move past this for consistency if the checks were easy to create, though. > Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/pat…

> I'm hesitant to cram into a time series database data that I consider boolean.

A boolean can be treated as a 1 or a 0. Sometimes you can even go richer than that, such as with a count of affected processes from your example - which you could convert back to a 1/0 if you wanted in the expression language.

> That architecture or configuration of this exporter isn't documented yet

The node_exporter as a whole still needs docs. That module is pretty new, and it's not exactly obvious what it does or why it exists.

Labels/dimensions are supported too, it's the same text format that Prometheus uses elsewhere ( http://prometheus.io/docs/instrumenting/exposition_formats/)

> if just magically sent numbers upstream from text files that I keep up to date

Pretty much, you'll also get the rest of the module of the node exporter (cpu, ram, disk, network) and associated consoles which http://www.boxever.com/monitoring-your-machines-with-prometh... describes how to setup.

Re: Prometheus: An open-source service monitoring system and time series database

#89

Post by bbrazil »

ithkuil wrote: post #87
jrv wrote: post #79

Earlier quoted context omitted.

An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout: [time series fingerprint : time range] -> [chunk of ts/value samples]] At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the content…

Did your previous leveldb approach perform way worse on reads, writes or both?

Both. When I switched our production Prometheis over, the consoles rendered more than twice as fast for a simple test case.

Re: Prometheus: An open-source service monitoring system and time series database

#90

Post by bbrazil »

simple10 wrote: post #86
bbrazil wrote: post #82

Earlier quoted context omitted.

> However, the pull/scraping model for stats could be problematic for larger scale. From experience of similar systems at massive scale, I expect no scaling problems with pulling in and of itself. Indeed, there's some tactical operational options you get with pull that you don't have with push. See http://www.boxever.com/push-vs-pull-for-monitoring for my general thoughts on the issue. > InfluxDB InfluxDB seems best…

Good point on push-vs-pull. I'm biased towards push because of microservices that behave like batch jobs. In effect, I'm using AMQP in a similar way as the Prometheus pushgateway. Agreed that InfluxDB is suited for event logging out of the box, but the March 2014 comparison of Influx is outdated IMO. I'm using Heka to send numeric time series data to Influx and full logs to ElasticSearch. It's possible to send full l…

> the March 2014 comparison of Influx is outdated IMO.

I think we expected that, feel free to add comments on the doc for things that are different now.

Post reply on HN