Systems Monitoring with Prometheus and Grafana
flightaware.engineering
Systems Monitoring with Prometheus and Grafana
1–10 of 107 posts
Re: Systems Monitoring with Prometheus and Grafana
#2Prometheus and Grafana are awesome, use them personally for all my monitoring.
However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using?
I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
Re: Systems Monitoring with Prometheus and Grafana
#3Prometheus is great. I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling "cloud native" projects that I probably didn't need or didn't have time to learn. There's actually a lot of adoption, you can find great exporters and grafana dashboards for almost any OSS you're running today. I started collecting metrics from Zookeeper and HBase in about an hour, having never had access to that telemetry before. From the existence of Cortex[1], it seems that Prometheus doesn't scale incredibly well, but I don't think many users will hit these limits.
Re: Systems Monitoring with Prometheus and Grafana
#4Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
Spark has worked decently for me if you need to be cloud agnostic.
Currently I’m in AWS land and Athena has been mostly working for what I need but I haven’t really pushed it that hard yet.
Re: Systems Monitoring with Prometheus and Grafana
#5Prometheus is great. I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling "cloud native" projects that I probably didn't need or didn't have time to learn. There's actually a lot of adoption, you can find great exporters and grafana dashboards for almost any OSS you're running today. I started collecting metrics from Zookeeper and HBase in about an hour, having never had…
My Prometheus system is a $10/mo Linode. It collects from 27 other hosts, and at least 100 services distributed across those hosts - doesn't even break a sweat. All the exporters run through a wireguard VPN. Prometheus is great for a small/medium SaaS type environment.
Re: Systems Monitoring with Prometheus and Grafana
#6Prometheus is great. I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling "cloud native" projects that I probably didn't need or didn't have time to learn. There's actually a lot of adoption, you can find great exporters and grafana dashboards for almost any OSS you're running today. I started collecting metrics from Zookeeper and HBase in about an hour, having never had…
Prometheus "scales" really well, but it does so via segmentation and federation, rather than increasing the size of an e.g. cluster. Some use cases don't fit to that model, so projects like Cortex and Thanos exist.
Re: Systems Monitoring with Prometheus and Grafana
#7Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
Just curious what your numbers are? Unique metrics, cardinality per metric, ingest rate, expected query ranges?
Re: Systems Monitoring with Prometheus and Grafana
#8Prometheus is great. I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling "cloud native" projects that I probably didn't need or didn't have time to learn. There's actually a lot of adoption, you can find great exporters and grafana dashboards for almost any OSS you're running today. I started collecting metrics from Zookeeper and HBase in about an hour, having never had…
Prometheus "scales" really well, but it does so via segmentation and federation, rather than increasing the size of an e.g. cluster. Some use cases don't fit to that model, so projects like Cortex and Thanos exist.
not vertically at least. the memory usage for indexing has room for improvement.
If I read the pprofs correctly, every scrape interval and every remote write allocates huge amounts of memory which is only cleaned up on garbage collection. You can easily need >64 gb ram for tenthousands of time series, otherwise you oom.
Re: Systems Monitoring with Prometheus and Grafana
#9Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
Just curious what your numbers are? Unique metrics, cardinality per metric, ingest rate, expected query ranges?
I have an instance that scrapes about 30K targets for 15 million metrics and it works better than you'd expect. The biggest performance issue we have is rendering the targets page.
We have a plan to split it down to less instances per node but it's worked well enough so far.
Re: Systems Monitoring with Prometheus and Grafana
#10Earlier quoted context omitted.
Prometheus "scales" really well, but it does so via segmentation and federation, rather than increasing the size of an e.g. cluster. Some use cases don't fit to that model, so projects like Cortex and Thanos exist.
not vertically at least. the memory usage for indexing has room for improvement. If I read the pprofs correctly, every scrape interval and every remote write allocates huge amounts of memory which is only cleaned up on garbage collection. You can easily need >64 gb ram for tenthousands of time series, otherwise you oom.
This has not been my experience at all. I'd file that as a bug.