Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prometheus: An open-source service monitoring system and time series database

prometheus.io

Post reply on HN
Print view View original

111–120 of 122 posts

Re: Prometheus: An open-source service monitoring system and time series database

#111

Post by pauldix »

jrv wrote: post #70
falcolas wrote: post #68

Earlier quoted context omitted.

So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code…

So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten…

InfluxDB CEO here. The data model for 0.9.0 should map up nicely with what you're trying to do. It supports measurement names, and tags. It would be great to work together to have InfluxDB as another option (other than OpenTSDB) for long term storage of metrics data.

Re: Prometheus: An open-source service monitoring system and time series database

#112

Post by pauldix »

jrv wrote: post #37
Rapzid wrote: post #28

Earlier quoted context omitted.

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

(Prometheus author here) Yeah, I did that benchmark with 11x overhead for storing typical Prometheus metrics in InfluxDB in March of 2014. Not sure if anything has changed conceptually since then, but if anyone can point out any flaws in my reasoning, that'd be interesting: https://docs.google.com/document/d/1OgnI7YBCT_Ub9Em39dEfx9Bu...

InfluxDB CEO here. The next version (0.9.0) has support for tags and efficiently encodes the measurement name + tagset as a single 4 byte uint.

Previously, you would have had to encode metadata in the series name. Otherwise if you used string columns you'd see a massive waste in disk space since they were repeated on every measurement.

Re: Prometheus: An open-source service monitoring system and time series database

#113

Post by pauldix »

jrv wrote: post #93
simple10 wrote: post #86

Earlier quoted context omitted.

Good point on push-vs-pull. I'm biased towards push because of microservices that behave like batch jobs. In effect, I'm using AMQP in a similar way as the Prometheus pushgateway. Agreed that InfluxDB is suited for event logging out of the box, but the March 2014 comparison of Influx is outdated IMO. I'm using Heka to send numeric time series data to Influx and full logs to ElasticSearch. It's possible to send full l…

Yeah, I would be really interested in hearing any arguments that would invalidate my research (because hey, if InfluxDB would actually be a good fit for long-term storage of Prometheus metrics, that'd be awesome, because it's Go and easy to operate). However, if the data model didn't change fundamentally (the fundamental InfluxDB record being a row containing full key/value metadata vs. Prometheus only appending a si…

InfluxDB CEO here. It's true that Bolt's performance is horrible if you're writing individual small data points. It gets orders of magnitude better if you batch up writes. The new architecture of InfluxDB allows us to safely batch writes without the threat of data loss if the server goes down before a flush (we have something like a write ahead log).

Basically, when the new version comes out, all new comparisons will need to be done because it's changing drastically.

Re: Prometheus: An open-source service monitoring system and time series database

#114

Post by jrv »

pauldix wrote: post #110
Rapzid wrote: post #28

Earlier quoted context omitted.

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

InfluxDB CEO here. With the current version of InfluxDB (0.8.8), metadata should be encoded in the name of the series, which means you don't repeat it. In the next version of InfluxDB (0.9.0), you can encode metadata as tags and it gets converted to a single id. With either of those schemes you should see much better numbers on storage.

Hey Paul, this is exciting news! So perhaps InfluxDB could work well as a Prometheus long-term storage backend after all. It would be so much nicer than having to run OpenTSDB based on Hadoop/HBase. It might be a while until I find the time, but looking forward to giving it another try!

Re: Prometheus: An open-source service monitoring system and time series database

#115

Post by jrv »

pauldix wrote: post #111
jrv wrote: post #70

Earlier quoted context omitted.

So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten…

InfluxDB CEO here. The data model for 0.9.0 should map up nicely with what you're trying to do. It supports measurement names, and tags. It would be great to work together to have InfluxDB as another option (other than OpenTSDB) for long term storage of metrics data.

Hey Paul, it'd be great to work together! Although long-term storage isn't a SoundCloud priority right now (meaning it's going to be more of a free-time project with less time allocated), it's definitely important for many others and it would be great to have. I think we should start with Prometheus->InfluxDB write support first, and then approach the more complicated problem of reading back from InfluxDB through Prometheus. What's the best place to get informed about the new 0.9.0 data model?

Re: Prometheus: An open-source service monitoring system and time series database

#116

Post by jrv »

jrv wrote: post #115
pauldix wrote: post #111

Earlier quoted context omitted.

InfluxDB CEO here. The data model for 0.9.0 should map up nicely with what you're trying to do. It supports measurement names, and tags. It would be great to work together to have InfluxDB as another option (other than OpenTSDB) for long term storage of metrics data.

Hey Paul, it'd be great to work together! Although long-term storage isn't a SoundCloud priority right now (meaning it's going to be more of a free-time project with less time allocated), it's definitely important for many others and it would be great to have. I think we should start with Prometheus->InfluxDB write support first, and then approach the more complicated problem of reading back from InfluxDB through Pro…

Ok, I found http://influxdb.com/blog/2014/12/08/clustering_tags_and_enha.... That all sounds very promising!

Re: Prometheus: An open-source service monitoring system and time series database

#117

Post by jrv »

jrv wrote: post #115
pauldix wrote: post #111

Earlier quoted context omitted.

InfluxDB CEO here. The data model for 0.9.0 should map up nicely with what you're trying to do. It supports measurement names, and tags. It would be great to work together to have InfluxDB as another option (other than OpenTSDB) for long term storage of metrics data.

Hey Paul, it'd be great to work together! Although long-term storage isn't a SoundCloud priority right now (meaning it's going to be more of a free-time project with less time allocated), it's definitely important for many others and it would be great to have. I think we should start with Prometheus->InfluxDB write support first, and then approach the more complicated problem of reading back from InfluxDB through Pro…

After reading http://influxdb.com/blog/2014/12/08/clustering_tags_and_enha..., I'm not completely clear on what "tags" mean. Are they single-word names, or are they conceptually key=value pairs like labels in Prometheus or tags in OpenTSDB?

EDIT: found https://github.com/influxdb/influxdb/pull/1059. Ok, does seem like tags mean key=value pairs.

Re: Prometheus: An open-source service monitoring system and time series database

#118

Post by ggambetta »

cespare wrote: post #17
[deleted] wrote: post #13

Earlier quoted context omitted.

[deleted]

You mean Borgmon? The Google monitoring system that stopped being a secret years and years ago? (One example from 2 years ago: https://www.reddit.com/r/IAmA/comments/177267/we_are_the_goo... )

Thank you. I did, in fact, make sure the name and the fact that it is a monitoring system were already public (and from an official source) before mentioning them in this post.

Re: Prometheus: An open-source service monitoring system and time series database

#119

Post by zeus13i »

After reading this thread and comparing Influx and Prometheus, I've concluded that both look promising. I was going to go with Prometheus (as it's easier to get started with), but I was really put off by the 'promdash' dashboard - it uses iframes and depends on mysql. So I'm going with InfluxDB + Grafana and I'll keep an eye out for developments.

Re: Prometheus: An open-source service monitoring system and time series database

#120

Post by bbrazil »

zeus13i wrote: post #119

After reading this thread and comparing Influx and Prometheus, I've concluded that both look promising. I was going to go with Prometheus (as it's easier to get started with), but I was really put off by the 'promdash' dashboard - it uses iframes and depends on mysql. So I'm going with InfluxDB + Grafana and I'll keep an eye out for developments.

You may want to look at the console templates, as they don't have any dependencies:

http://prometheus.io/docs/visualization/consoles/

Post reply on HN