Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prometheus: An open-source service monitoring system and time series database

prometheus.io

Post reply on HN
Print view View original

61–70 of 122 posts

Re: Prometheus: An open-source service monitoring system and time series database

#61

Post by beorn7 »

falcolas wrote: post #57
mbell wrote: post #56

From the storage system docs: > which organizes sample data in chunks of constant size (1024 bytes payload). These chunks are then stored on disk in one file per time series. That is concerning, is this going to have the same problem with disk IO that graphite does? i.e. Every metric update requires a disk IO due to this one file per metric structure.

This was my first thought as well. Having one inode per metric measured can easily overwhelm some file systems, and the IO overhead on those disks gets silly (especially if it's not a SSD capable of constant-time writes against sparse data). Combine with the Cacti pull model, and I think a wait-and-see attitude is the best for this for now.

We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.

Re: Prometheus: An open-source service monitoring system and time series database

#62

Post by bbrazil »

wyldfire wrote: post #59
bbrazil wrote: post #53

Earlier quoted context omitted.

We support that use case, here's a guide: http://www.boxever.com/monitoring-your-machines-with-prometh...

That's great! > For machine monitoring Prometheus offers the Node exporter Is it possible for the frontend to utilize data from the cron-invoked sar/sadc that already covers much of this data? http://sebastien.godard.pagesperso-orange.fr/

The default instrumentation that comes with node_exporter covers far more than what systat provides. Retrieving data from /proc at every scrape also gives you a more accurate timestamp, which helps reduce graph artifacts.

As an aside, if you have machine-level cronjobs you want to expose metrics from you can use the textfile[1] module of the node_exporter, which reads in data from *.prom files in the same format as accepted by the Pushgateway.

[1]https://github.com/prometheus/node_exporter/blob/master/coll... [2]https://github.com/prometheus/pushgateway

Re: Prometheus: An open-source service monitoring system and time series database

#63

Post by jrv »

bhuga wrote: post #55

It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted for monitoring, notifications or graphing as nginx is for a web server. Not that I'd do a better job, but every time I further configure our monitoring system, I get that feeling that we're missing something as an industry. It's a space with lots of tools tha…

Brian already hinted at this, but the main reason why we do this is to have coherency of the data model across the whole chain:

client app -> Prometheus server -> alerting rules -> alert handling

This allows e.g. initial dimensional labels from the clients to cleanly propagate all the way into alerts, and allows you to silence, aggregate, or route alerts by any label combination.

If OpenTSDB-style dimensional metrics would have been more of a standard already, maybe that would be different.

Re: Prometheus: An open-source service monitoring system and time series database

#64

Post by SEJeff »

As a graphite maintainer, see my other post about problems with graphite:

https://news.ycombinator.com/item?id=8908423

I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform.

The more OSS metrics solutions, the better!

Re: Prometheus: An open-source service monitoring system and time series database

#66

Post by kordless »

clarkevans wrote: post #25

We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't need access to every single producer. I see the pushgateway, but it seems deliberately not a centralized storage. I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metr…

Is the use case mostly about permissions and networking setup? I could see why a pull model wouldn't work in those cases.

Re: Prometheus: An open-source service monitoring system and time series database

#67

Post by bbrazil »

SEJeff wrote: post #64

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solu…

Great! We're over in #prometheus on Freenode if there's anything we can help you with.

Re: Prometheus: An open-source service monitoring system and time series database

#68

Post by falcolas »

beorn7 wrote: post #61
falcolas wrote: post #57

Earlier quoted context omitted.

This was my first thought as well. Having one inode per metric measured can easily overwhelm some file systems, and the IO overhead on those disks gets silly (especially if it's not a SSD capable of constant-time writes against sparse data). Combine with the Cacti pull model, and I think a wait-and-see attitude is the best for this for now.

We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.

So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me.

> file systems are way better in managing the data

Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4.

After having briefly dug into the code, I'm particularly worried about the fact that instead of minimizing iops by only writing the relevant changes to the same file, Prometheus is constantly copying data from one file to the next, both to perform your checkpoints and to invalidate old data. That's a lot of iops for such basic (and frequently repeated) tasks.

Still in wait-and-see mode.

Re: Prometheus: An open-source service monitoring system and time series database

#69

Post by bhuga »

jrv wrote: post #63
bhuga wrote: post #55

It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted for monitoring, notifications or graphing as nginx is for a web server. Not that I'd do a better job, but every time I further configure our monitoring system, I get that feeling that we're missing something as an industry. It's a space with lots of tools tha…

Brian already hinted at this, but the main reason why we do this is to have coherency of the data model across the whole chain: client app -> Prometheus server -> alerting rules -> alert handling This allows e.g. initial dimensional labels from the clients to cleanly propagate all the way into alerts, and allows you to silence, aggregate, or route alerts by any label combination. If OpenTSDB-style dimensional metrics…

I haven't used Prometheus yet, but I'm not sure I agree that it makes sense to send dimensional time series all the way down the alert chain. It allows full-featured alerts to be used, yes. But you're not going to be able to put all of your alert logic into the prometheus chain, so you're going to end up having alert rules in at least two places.

For example, in response to the latest security hullabaloo, we recently added a Nagios check for libc being too old, and for running processes that are linked to a libc older than the one on disk. This isn't a time series check; it's just an alert condition. Polling for that alert condition isn't what Prometheus is going to be good at, so cramming it into that alert pipeline is awkward. If we used something like Prometheus, we'd have rules in two places already.

Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There's still some logic about what you're measuring in Prometheus, but not about what thresholds are alerts, or when, or for how long, or who cares. Threshold rules could all live in one place.

This isn't a knock on Prometheus, which of course supports a workflow along those lines, and seems interesting. Nor is it a knock on the prometheus alert system. I just wonder why software in the monitoring/notification/alerting space always ends up muddy. I believe the overlapping boundaries between projects are a symptom, not a cause. I keep asking myself if there's some fundamental complexity that I'm not yet appreciating.

Re: Prometheus: An open-source service monitoring system and time series database

#70

Post by jrv »

falcolas wrote: post #68
beorn7 wrote: post #61

Earlier quoted context omitted.

We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.

So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code…

So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though.

RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten by new ones at predictable periods. It's great because you can just derive the file position of a sample based on its timestamp, but in Prometheus, samples can have arbitrary time stamps and gaps between them, and time series can also grow arbitrarily large (depending on currently configured retention period) and our data format needs to support that.

InnoDB: not sure how this works internally, but given it's usually used in MySQL, does it work well for time series data? I.e. millions of time series that each get frequent appends?

KairosDB: depends on Cassandra, AFAICS. One main design goal was to not depend on complex distributed storage for immediate fault detection, etc.

InfluxDB: looks great, but has an incompatible data model. See http://prometheus.io/docs/introduction/comparison/#prometheu...

I guess a central question that touches on your iops one is: you always end up having a two-dimensional layout on disk: timeseries X time. I don't really see a way to both store and retrieve data in such a way that you can arbitrarily select a time range and time series without incurring a lot of iops either on read or write.

Post reply on HN