Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systems Monitoring with Prometheus and Grafana

flightaware.engineering

Post reply on HN
Print view View original

61–70 of 107 posts

Re: Systems Monitoring with Prometheus and Grafana

#61

Post by cmckn »

jeffbee wrote: post #41
cmckn wrote: post #3

Prometheus is great. I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling "cloud native" projects that I probably didn't need or didn't have time to learn. There's actually a lot of adoption, you can find great exporters and grafana dashboards for almost any OSS you're running today. I started collecting metrics from Zookeeper and HBase in about an hour, having never had…

Interesting to say that Prometheus is "fledgling". The project is almost 7 years old and the Google thing on which it is based is ~15 years old.

> I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling...

I didn’t know the age of the project, because I hadn’t heard of it. That’s why I go on to say that in actuality it has a ton of adoption and I’ve had a great experience with it.

Re: Systems Monitoring with Prometheus and Grafana

#62

Post by jdub »

halfmatthalfcat wrote: post #2

Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?

Honeycomb https://honeycomb.io/ is laser focused on this stuff. They built their own datastore (similar to Druid but schemaless) so they could create the experience they were aiming for.

They talk a lot about collaborative troubleshooting, and the user interface reflects that. It's actually fun (?!) to drill down from heatmaps to individual events with Honeycomb's little comparison charts lighting the way.

Re: Systems Monitoring with Prometheus and Grafana

#63

Post by MetalMatze »

bacondude3 wrote: post #34

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

Prometheus supports writing (replicating) data to a remote endpoint on a per scrape basis with a protocol called remote-write. You can pretty easily set that up on any Prometheus instance. There are quite some implementations to receive those remote-write requests: https://prometheus.io/docs/operating/integrations/#remote-en...

You're probably exactly looking for something like that. In fact, I've given a talk about a similar scenario at the KubeCon San Diego: https://www.youtube.com/watch?v=FrcfxkbJH20

Disclosure: I work on Thanos and Thanos Receiver which implements that protocol.

Re: Systems Monitoring with Prometheus and Grafana

#64

Post by dxhdr »

sagichmal wrote: post #19
site-packages1 wrote: post #12

What do you all do with the collected metrics over time? Do you store everything forever, drop everything after a couple weeks, or something on between? I've heard of people thinning out old data a bit (?) and storing it long term rather than storing everything. What's the usual thing people do?

High fidelity operational metrics have a useful half life measured in days or weeks. Read patterns for longer term use cases are also categorically different. Best architecture is to have a separate system for long term stuff, which treats Prometheus as a data source. Then Prometheus can drop after 14-28d.

> High fidelity operational metrics have a useful half life measured in days or weeks.

It depends on how high fidelity you're talking but in my experience retaining these metrics can be valuable, not only for viewing seasonal trends already mentioned in another reply but for debugging problems. It can be helpful to be able to view prior events and compare metrics at those times to a current scenario, for example as a part of a postmortem analysis. I do agree that the usefulness of old metrics falls off with time. Metrics issued from a system 3 years ago likely have little in common with the system running today.

Re: Systems Monitoring with Prometheus and Grafana

#65

Post by cassianoleal »

bacondude3 wrote: post #34

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

I've been using ZeroTier [0] so whenever my laptop is online, it's also accessible by Prometheus so it can be scraped. I run ZeroTier in a container on my router, and on each "road warrior" that needs LAN access (laptop, phone...).

The caveat is that I have no metrics when the laptop is offline but that doesn't happen very often anyway.

[0] https://www.zerotier.com/

Re: Systems Monitoring with Prometheus and Grafana

#66

Post by codeduck »

site-packages1 wrote: post #12

What do you all do with the collected metrics over time? Do you store everything forever, drop everything after a couple weeks, or something on between? I've heard of people thinning out old data a bit (?) and storing it long term rather than storing everything. What's the usual thing people do?

7 day retention in Prometheus, pushing to something like VictoriaMetrics for downsampling and long term storage. Prometheus is great for collection but rubbish for managing large data sets

Re: Systems Monitoring with Prometheus and Grafana

#67

Post by jpgvm »

Nihilartikel wrote: post #27
halfmatthalfcat wrote: post #2

Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?

I've used druid.io in the past and it had worked well, but it's a lot of trouble to set up and tune.. Haven't tried it, but clickhouse looks good and has approximate aggregations for high cardinality dimensions.

Druid truly is still king in this space. The setup has become less onerous over time. It handles arbitrarily high cardinality and dimensionality with ease and its support for sketching algorithms leaves other similar systems (especially Prometheus) in the dust.

Re: Systems Monitoring with Prometheus and Grafana

#68

Post by skohan »

dijit wrote: post #14

Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…

> Grafana truly is best in class

Really? Recently we've been playing with Chronograf with InfluxDB and most people find it a lot nicer to work with than Grafana (specifically because it makes discoverability a lot nicer)

Re: Systems Monitoring with Prometheus and Grafana

#69

Post by _xrjp »

skohan wrote: post #68
dijit wrote: post #14

Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…

> Grafana truly is best in class Really? Recently we've been playing with Chronograf with InfluxDB and most people find it a lot nicer to work with than Grafana (specifically because it makes discoverability a lot nicer)

For our modest cloud infra, InfluxData TICK (InfluxDB, Kapacitor, Chronograf and Telegraf) stack has fitted exactly with our needs. We really like its folding building-blocks, interoperability and yeah... easy discoverability and configuration. But also its very convenient InfluxQL which lets us customize reports with ease on InfluxDB.

Re: Systems Monitoring with Prometheus and Grafana

#70

Post by heliodor »

bjakubski wrote: post #21
site-packages1 wrote: post #12

What do you all do with the collected metrics over time? Do you store everything forever, drop everything after a couple weeks, or something on between? I've heard of people thinning out old data a bit (?) and storing it long term rather than storing everything. What's the usual thing people do?

Depends on your needs really. Some metrics we do (for now) keep indefinitely. We're using Thanos to ship data to bucket in object storage Some metrics we do keep for two weeks only.

Then that's business data not monitoring data. Whole different use case and tools. One loses its value over time, the other doesn't.
Post reply on HN