Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

21–30 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#21
post #3

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

We had major issues with scaling InfluxDB. We use clickhouse (graphite table engine) now and it is more than order of magnitude more resource efficient.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#22
I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome.

I'm now having better results and experience storing data in ClickHouse (yes, not a timeseries dB).

From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising.

It might just be me.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#23
post #3

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

Influx is really shitting the bed with how they're handling the InfluxDB 2.0 release. The docs are a mess, and the migration tool seems like the result of a weekend hackathon. They're leaving a lot of customers with long term metrics in a tough spot. If you're thinking about using Influx for long term data storage, look elsewhere. The company continuously burns customer goodwill by going against the grain, and buckin…

(I'm the VPE at Influxdata).

I appreciate this sentiment. We've been focused on a building a SaaS version of Influxdata and are committed to a paired open source version of that. The open source version has been lagging as we work on the SaaS side.

However, we are committed to shipping a GA version of the OSS 2.0 stack around the end of Q3 that offers an in-place data migration capability from 1.x OSS.

We've spoken about this publicly in other forums. You can google "influxdays London talks" to hear Paul Dix (CTO/Founder) talk more about our OSS plans.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#24
I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this?

I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The monitor server is just a DO droplet and our collectors require zero overhead.

The specs list and setup costs in time and complexity are steep with a Grafana stack - is there any value besides just the visual? I know they have the ability to do all manner of custom plugins, dashboards, etc, but if you just care about the good stuff (uptime+performance), what does Grafana give you that rsync'ing sar data can't?

PS: we have a graphical parser of the data written using python and matplotlib. very lightweight, and we also get pretty graphs to print and give to upstairs.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#25

Earlier quoted context omitted.

Telegraf and collectd do roughly the same thing. They run some plugins to get data and push the results to a given metrics sink. I asked because TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) is a known solution and a fairly standard way to add elements of monitoring to your system.

> TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) is a known solution and a fairly standard way to add elements of monitoring to your system. Amusing that I've never heard of any of this but have heard and used collectd. It's obviously no where near as common as a lamp stack or anywhere near common at all, so asking why it over something else is answered by "someone made it up so it's better".

Collectd is only one part here. Did you need a solution for inline processing/aggregation and alerting? If not, you wouldn't run into TICK. It's not common overall, because few environments need to go that far.

I don't get the comparison to LAMP popularity. Insects are more common than cars too. They're different things ¯\_(ツ)_/¯

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#26
I found the software in this stack to be very bloated and difficult to maintain. Large, complicated software has a tendency to fall flat on its face when something goes wrong, and this is a domain where reliability is paramount. I wrote about a different approach (Prometheus + Alertmanager) for sourcehut, if you're curious:

https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#27
InfluxDB and Grafana worked great for us when I created a live monitoring system for a fleet of prototype test robots. It was simple to set up new data streams. We started with Graphite but switched to InfluxDB for it's flexibility (Grafana works with both!)

I would add to the guide that you need to be careful about formatting the lines into InfluxDB because where you out the space and commas determines what is indexed or not! Also data types should be specific (ie make sure you are setting integer vs float correctly).

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#29
post #22

I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome. I'm now having better results and experience storing data in ClickHouse…

I have also had scalability and reliability issues with influx. And full of silly limitations like tagset cardinality and not being able to delete points in a specific retention policy etc. Am moving to classic rdbms and timescaledb.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#30

I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this? I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The mon…

The biggest advantage is the near-real-time aspect of this. During an outage, having live metrics is essential. Does your custom system allow you to see live metrics as they happen, or do you need to re-run your aggregation scripts every 5 minutes?
Post reply on HN