I've done this in my company with about 100 servers using KairosDB + Grafana and CollectD...
If I were to do it again I'd probably use Grafana and Prometheus
Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
51–60 of 298 posts
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#52Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.
I'd be interested to know a bit of detail as we're looking into Grafana/Promethus a bit.
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#53Nothing against choosing this set of apps really, but I'm curious why collectd and not telegraf which does the same kind of metrics probes and is a part of the TICK stack.
Although I've successfully used collectd in the past, I would choose telegraf over it now. This is especially true if I'm sending data to InfluxDB,s ince it and telegraf are primarily developed by InfluxData.
Both telegraf and collectd are mature programs with a large base of well-documented plugins. However, telegraf development is more active, and I find its golang codebase much more approachable than the C codebase of collectd. Obviously, YMMV.
In general though, I wouldn't advise adopting the entire TICK stack. Chronograf is inferior to Grafana. Kapacitor has a super-steep learning curve and is being deprecated by InfluxData. TIG (telegraf, influxdb, grafana) can be a good choice. Prometheus plus Grafana is worth evaluating as well; there's arguably more momentum in that ecosystem than in InflxuDB's.
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#54Earlier quoted context omitted.
One point of note for SourceHut's Prometheus use is that we generally don't make dashboards. I don't really like Grafana. I will sometimes use gnuplot with styx to plot graphs on an as-needed basis: https://github.com/go-pluto/styx This is how I made the plots in that blog post.
Can't you generate the same kind of graphs you have there with the normal Prometheus query explorer / web ui?
On a basic level, yes, but I often just use it as a starting point for more complex gnuplot graphs, or different kinds of visualizations - box plots, histograms, etc.
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#55I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome. I'm now having better results and experience storing data in ClickHouse…
I have also had scalability and reliability issues with influx. And full of silly limitations like tagset cardinality and not being able to delete points in a specific retention policy etc. Am moving to classic rdbms and timescaledb.
Indeed, cardinality limit is a very painful aspect which blocked us since day one for certain metrics. I confess, at the time I didn't know any better. Now I wouldn't recommend it under any circumstances.
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#56Earlier quoted context omitted.
(I'm the VPE at Influxdata). I appreciate this sentiment. We've been focused on a building a SaaS version of Influxdata and are committed to a paired open source version of that. The open source version has been lagging as we work on the SaaS side. However, we are committed to shipping a GA version of the OSS 2.0 stack around the end of Q3 that offers an in-place data migration capability from 1.x OSS. We've spoken a…
Classic "two masters" problem.
This was from the CTO last week: "This work won't be landing anywhere until sometime next year and it'll be landing in our Cloud 2 offering first." So the OSS is definitely a second-class citizen. And now that they've dropped all DevRel activity, don't expect much attention for OSS Developers and users.
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#57I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this? I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The mon…
I'm not experienced with the CollectD stack, but I use Prometheus + Grafana to monitor probes. My two cents: - Fairly lightweight. Prometheus deals with quite a lot of series without much memory or CPU usage. - Integration with a lot of applications. Prometheus lets me monitor not only the system, but other applications such as Elastic, Nginx, PostgreSQL, network drivers... Sometimes I need an extra exporter, but the…
Author here. I'll probably write another tutorial focusing on Prometheus, instead of CollectD.
Thanks for suggestion
SerHack
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#58[deleted]
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#59Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.
This is good advice, however you will also want to make sure you have a plan to get off datadog when you grow. Datadog is one of the easiest to use and most comprehensive out of the box. But it gets really expensive as you begin to scale up and add servers, cloud accounts, services etc....
At a certain scale, rolling your own monitoring and alerting becomes cost effective again as Datadog begins to charge an arm and a leg. I've seen Datadog bills that could easily pay for 2 full time engineers.
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#60Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.
I'd be interested to know a bit of detail as we're looking into Grafana/Promethus a bit.
Prometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.