Prometheus: An open-source service monitoring system and time series database • Hacker News Board

Re: Prometheus: An open-source service monitoring system and time series database

#91

Post by KyleBrandt » Wed, Feb 04, 2015, 7:05 PM UTC

jrv wrote: post #85

KyleBrandt wrote: post #83

Earlier quoted context omitted.

Shameless plug. But Bosun's focus is largely on Alerting ( http://bosun.org ). We have an expression language built in that allows for complex rules. It leverages OpenTSDBs multi dimensional facet to make alert instantiations, but you can also change the scope by transposing your results. It also now supports Logstash and Graphite as backends as well. The Graphite support is thanks to work at Vimeo. Another nice thin…

Great. Yeah, adding unit tests for alerts is something that we still need to add the capability for. But at least you can always already manually graph your alert expressions and see how they would have evaluated over time. Interesting work on Bosun by the way! Seems like there is quite some overlap with Prometheus, but I yet have to study it in depth. Is my impression correct that OpenTSDB is a requirement, or is th…

OpenTSDB is our main backend. But we can also query graphite and logstash. However the graphing page doesn't work with Graphite.

Re: Prometheus: An open-source service monitoring system and time series database

#92

Post by simple10 » Wed, Feb 04, 2015, 7:26 PM UTC

bbrazil wrote: post #90

simple10 wrote: post #86

Earlier quoted context omitted.

Good point on push-vs-pull. I'm biased towards push because of microservices that behave like batch jobs. In effect, I'm using AMQP in a similar way as the Prometheus pushgateway. Agreed that InfluxDB is suited for event logging out of the box, but the March 2014 comparison of Influx is outdated IMO. I'm using Heka to send numeric time series data to Influx and full logs to ElasticSearch. It's possible to send full l…

> the March 2014 comparison of Influx is outdated IMO. I think we expected that, feel free to add comments on the doc for things that are different now.

Cool. Will do once InfluxDB 0.9 is released in March. Not worth comparing to 0.8 since so much is changing under the hood.

Re: Prometheus: An open-source service monitoring system and time series database

#93

Post by jrv » Wed, Feb 04, 2015, 7:28 PM UTC

simple10 wrote: post #86

bbrazil wrote: post #82

Earlier quoted context omitted.

> However, the pull/scraping model for stats could be problematic for larger scale. From experience of similar systems at massive scale, I expect no scaling problems with pulling in and of itself. Indeed, there's some tactical operational options you get with pull that you don't have with push. See http://www.boxever.com/push-vs-pull-for-monitoring for my general thoughts on the issue. > InfluxDB InfluxDB seems best…

Good point on push-vs-pull. I'm biased towards push because of microservices that behave like batch jobs. In effect, I'm using AMQP in a similar way as the Prometheus pushgateway. Agreed that InfluxDB is suited for event logging out of the box, but the March 2014 comparison of Influx is outdated IMO. I'm using Heka to send numeric time series data to Influx and full logs to ElasticSearch. It's possible to send full l…

Yeah, I would be really interested in hearing any arguments that would invalidate my research (because hey, if InfluxDB would actually be a good fit for long-term storage of Prometheus metrics, that'd be awesome, because it's Go and easy to operate).

However, if the data model didn't change fundamentally (the fundamental InfluxDB record being a row containing full key/value metadata vs. Prometheus only appending a single timestamp/value sample pair for an existing time series whose metadata is only stored and indexed once), I wouldn't expect the outcome to be qualitatively different except that the exact storage blowup factor will vary.

Interesting to hear that InfluxDB is using BoltDB now. I benchmarked BoltDB against LevelDB and other local key-value stores around a year ago, and for a use case of inserting millions of small keys, it took 10 minutes as opposed to LevelDB taking a couple of seconds (probably due to write-ahead-log etc.). So BoltDB was a definite "no" for storing the Prometheus indexes. Also it seems that the single file in which BoltDB stores its database never shrinks again when removing data from it (even if you delete all the keys). That would also be bad for the Prometheus time series indexing case.

Re: Prometheus: An open-source service monitoring system and time series database

#94

Post by jrv » Wed, Feb 04, 2015, 7:46 PM UTC

KyleBrandt wrote: post #91

jrv wrote: post #85

Earlier quoted context omitted.

Great. Yeah, adding unit tests for alerts is something that we still need to add the capability for. But at least you can always already manually graph your alert expressions and see how they would have evaluated over time. Interesting work on Bosun by the way! Seems like there is quite some overlap with Prometheus, but I yet have to study it in depth. Is my impression correct that OpenTSDB is a requirement, or is th…

OpenTSDB is our main backend. But we can also query graphite and logstash. However the graphing page doesn't work with Graphite.

Ah, ok. By the way, in case you make it to GopherCon this year, it would be interesting to exchange ideas! One of the things I'm happy about is that finally systems with multi-dimensional time series (based on OpenTSDB or not) are becoming more common...

Re: Prometheus: An open-source service monitoring system and time series database

#95

Post by mdeeks » Wed, Feb 04, 2015, 8:12 PM UTC

SEJeff wrote: post #64

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solu…

A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend-.cpu-0.cpu-used Where "frontend-" expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Would love to know if you have ideas. Is this a limitation of graphite-web itself?

I have large graphite install of 20+ graphite carbon nodes running on SSDs and three additional graphite-web instances in front generating graphs. Ingesting something like 1 million metrics/min.

Also I didn't realize there were still graphite maintainers (seriously. not trolling). There hasn't been a release of graphite in well over a year. I assumed it was dead by now. Any idea when we'll get a fresh release?

Re: Prometheus: An open-source service monitoring system and time series database

#96

Post by 0xdeadbeefbabe » Wed, Feb 04, 2015, 8:21 PM UTC

While monitoring is obviously useful, I'm not understanding the obvious importance of a time series database. Can you collect enough measurements for the time series database to be useful? I worry that I would have lots of metrics to backup my wrong conclusions. I also worry that so much irrelevant data would drown out the relevant stuff, and cause the humans to ignore the system in time. I work with computers and servers, and not airplanes or trains.

Re: Prometheus: An open-source service monitoring system and time series database

#97

Post by SEJeff » Wed, Feb 04, 2015, 8:38 PM UTC

mdeeks wrote: post #95

SEJeff wrote: post #64

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solu…

A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend- .cpu-0.cpu-used Where "frontend- " expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Wo…

We are in the final stages of the last 0.9.x release, 0.9.13. From then on, we're going to be making some more noticeable changes that break some backwards compat to make the project a lot more pleasant.

Note that 0.9.13 is almost ready to be cut: https://github.com/graphite-project/graphite-web/commit/7862...

https://github.com/graphite-project/carbon/commit/e69e1eb59a...

https://github.com/graphite-project/whisper/commit/19ab78ad6...

Anything in the master branch is what will be in 0.10.0 when we're ready to cut that. I think we'll spend some more cycles in 0.10.x focusing on non-carbon / non-whisper / non-ceres backends that should allow much better scalability. Some of these include cassandra, riak, etc.

For it timing out, it is a matter of general sysadmin spleunking to figure out what is wrong. It could be IO on your carbon caches, or CPU in your render servers (where it uses cairo). I'm a HUGE fan of grafana for doing 100% of the dashboards and only using graphite-web to spit out json, or alternatively to use graphite-api.

Take a look at the maxDataPoints argument to see if that will help your graphs to not timeout however.

Re: Prometheus: An open-source service monitoring system and time series database

#98

Post by bbrazil » Wed, Feb 04, 2015, 9:14 PM UTC

0xdeadbeefbabe wrote: post #96

While monitoring is obviously useful, I'm not understanding the obvious importance of a time series database. Can you collect enough measurements for the time series database to be useful? I worry that I would have lots of metrics to backup my wrong conclusions. I also worry that so much irrelevant data would drown out the relevant stuff, and cause the humans to ignore the system in time. I work with computers and se…

> Can you collect enough measurements for the time series database to be useful?

Yes, instrument everything. See http://prometheus.io/docs/practices/instrumentation/#how-to-...

> I worry that I would have lots of metrics to backup my wrong conclusions.

This is not so much a problem with time series as a question of epistemology. Well chosen consoles will help your initial analysis, and after that it's down to correct application of the scientific method.

> I also worry that so much irrelevant data would drown out the relevant stuff

I've seen many attempts by smart people to try and do automatic correlation of time series to aid debugging. It's never gotten out of the toy stage, as there is too much noise. You need to understand your metrics in order to use them.

Re: Prometheus: An open-source service monitoring system and time series database

#99

Post by bbrazil » Wed, Feb 04, 2015, 9:19 PM UTC

mdeeks wrote: post #95

SEJeff wrote: post #64

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solu…

A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend- .cpu-0.cpu-used Where "frontend- " expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Wo…

For comparison I tried out >1k cpu plots in Prometheus on a m1.large with 2xHDDs. It took 20s with a cold cache.

Re: Prometheus: An open-source service monitoring system and time series database

#100

Post by mdeeks » Wed, Feb 04, 2015, 9:28 PM UTC

bbrazil wrote: post #99

mdeeks wrote: post #95

Earlier quoted context omitted.

A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend- .cpu-0.cpu-used Where "frontend- " expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Wo…

For comparison I tried out >1k cpu plots in Prometheus on a m1.large with 2xHDDs. It took 20s with a cold cache.

I'd be interested in hearing how it performs when rendering a huge page of graphs each with dozens to hundreds of graph lines.

Unfortunately though Prometheus lacks easy horizontal scaling just like Graphite. It sounds like Prometheus is worse actually since it mentions manual sharding rather than consistent hashing that Graphite does. This rules out Prometheus as an alternative to Graphite for me even if it does render complex graphs better. I'm definitely keeping my eye on this one though.