Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prometheus: An open-source service monitoring system and time series database

prometheus.io

Post reply on HN
Print view View original

31–40 of 122 posts

Re: Prometheus: An open-source service monitoring system and time series database

#31

Post by bbrazil »

Rapzid wrote: post #28
clarkevans wrote: post #25

We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't need access to every single producer. I see the pushgateway, but it seems deliberately not a centralized storage. I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metr…

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

We favour pull, but we don't force it. The collectd exporter is an integration with a push-based system for example.

My full thoughts are up at http://www.boxever.com/push-vs-pull-for-monitoring The short version is that I consider pull slightly better, but not majorly so. Push is more difficult to work with as you scale up, but it is doable.

Re: Prometheus: An open-source service monitoring system and time series database

#33

Post by ithkuil »

lifeisstillgood wrote: post #20
[deleted] wrote: post #18

Earlier quoted context omitted.

[deleted]

What on earth? Google imposes non disclosure on ex-employees to mention by name in-house monitoring systems? (Presumably Borg-monitor?) That strikes me as a bit paranoid, not letting the name of a monitoring system be revealed. Am I missing something?

Once I heard a urban legend about a programmer being sued because he was using mickeyMouse as a variable name.

I wouldn't be surprised if many of such cases of non-disclosure are to avoid hungry lawyers for trying to suck some blood out of rich companies. I guess the star trek franchise is owned by Paramount or something like this.

Re: Prometheus: An open-source service monitoring system and time series database

#34

Post by Rapzid »

clarkevans wrote: post #30
Rapzid wrote: post #28

Earlier quoted context omitted.

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

What do you use for push-oriented time-series metrics? We are thinking of using LogStash (it has a go client). Unfortunately, as I understand, it's not time-series oriented.

I push all my logs into logstash->elasticsearch.

For metrics I go Riemann->Graphite. Riemann comes with a graphite compatible server so I push straight to that for processing and alerting. I also send from Riemann to logstash and logstash to Riemann where it makes sense.

For my metrics dashboard I use Grafana which is really awesome. I make use of its templating pretty heavily as I run a lot of individual IIS clusters. I can create cluster overview and drill down dashboards and template it so I can just change the cluster number to see its stats. You can also link directly into a dashboard passing variables as a query string parameter. Pretty excellent.

Re: Prometheus: An open-source service monitoring system and time series database

#35

Post by bbrazil »

clarkevans wrote: post #30
Rapzid wrote: post #28

Earlier quoted context omitted.

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

What do you use for push-oriented time-series metrics? We are thinking of using LogStash (it has a go client). Unfortunately, as I understand, it's not time-series oriented.

https://github.com/rcrowley/go-metrics is one I've came across while researching potential Prometheus integrations, it seems to support a number of backends including Graphite and InfluxDB.

One challenge with that model of metrics is that it assumes that the monitoring system doesn't have much ability to work with samples, so it makes up for this by calculating rates and quantiles in the client. This isn't as accurate as doing it in the server, and some instrumentation systems don't allow you to extract out the raw data to allow you to do it in the server.

For example say you've a 1-minute rate exported/scraped every minute. If you miss a push/scrape you lose all information about that minute. Similarly if you're delayed by a second, you'll miss a spike just after the previous push/scrape. If instead you expose a monotonically increasing counter and calculate a rate over that in the server, you don't lose that data.

Re: Prometheus: An open-source service monitoring system and time series database

#36

Post by hartror »

clarkevans wrote: post #27
bbrazil wrote: post #26

Earlier quoted context omitted.

(One of the Prometheus authors here) My understanding is that InfluxDB will work for your use case, as it's all push. > I see the pushgateway, but it seems deliberately not a centralized storage. Yeah, the primary use case for the pushgateway is service-level metrics for batch jobs. May I ask why accessing your producers is a problem? I know for Storm I was considering writing a plugin that'd hook into it's metrics s…

Each node in our network has it's own access control permissions and organizational owner. It's easy enough to provide a centralized service with an end-point to push statistics from each node. In this case, organizational participation is "distributed". A pull model, while technically distributed, is organizationally centralized: I have to get each node's owner to grant me direct access. Politically, and for securit…

This is a common problem that you describe.

Re: Prometheus: An open-source service monitoring system and time series database

#37

Post by jrv »

Rapzid wrote: post #28
clarkevans wrote: post #25

We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't need access to every single producer. I see the pushgateway, but it seems deliberately not a centralized storage. I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metr…

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

(Prometheus author here)

Yeah, I did that benchmark with 11x overhead for storing typical Prometheus metrics in InfluxDB in March of 2014. Not sure if anything has changed conceptually since then, but if anyone can point out any flaws in my reasoning, that'd be interesting:

https://docs.google.com/document/d/1OgnI7YBCT_Ub9Em39dEfx9Bu...

Re: Prometheus: An open-source service monitoring system and time series database

#38

Post by jrv »

clarkevans wrote: post #25

We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't need access to every single producer. I see the pushgateway, but it seems deliberately not a centralized storage. I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metr…

(initial Prometheus author here)

I wouldn't totally rule out the pushgateway for this use case. If we decided to implement a metrics timeout in the pushgateway (https://github.com/prometheus/pushgateway/issues/19), this would also take care of stale metrics from targets that are down or decommissioned. The pushing clients would probably even want to set a client-side timestamp in that case, as they are expected to be pushing regularly enough for the timestamp to not become stale (currently Prometheus considers time series stale that are >5 minutes old by default, see also the "Improved staleness handling" item in http://prometheus.io/docs/introduction/roadmap/.

Re: Prometheus: An open-source service monitoring system and time series database

#39

Post by jrv »

secure wrote: post #9

I used to use InfluxDB + a custom program to scrape HTTP endpoints and insert them into InfluxDB before. After playing around with Prometheus for a day or so, I’m convinced I need to switch to Prometheus :). The query language is so much better than what InfluxDB and others provide.

(Prometheus author here)

Thanks, that's awesome to hear! Feel free to also join us on #prometheus on freenode or our mailing list: https://groups.google.com/forum/#!forum/prometheus-developer...

Re: Prometheus: An open-source service monitoring system and time series database

#40

Post by tinco »

In your architecture I see a single monolithic database server called 'Prometheus'. Does it shard? I can't find it in the documentation. You mention it's compatible with TSDB, why did you choose to implement your own backend, or is this a fork of TSDB?

The tech does look awesome though!

Post reply on HN