Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

291–298 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#291
post #289

Does anyone have experience with an InfluxDB-based netdata setup? When we evaluated multiple options, we found that netdata does a lot of the heavy lifting. What are the advantages of a CollectD-based setup?

Well, that's the beauty of Netdata, no need to configure collectors. As for InfluxDB integration, it's fairly easy to export from Netdata using what InfluxDB calls the line protocol format (the "plaintext interface").

https://learn.netdata.cloud/docs/agent/exporting

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#292
post #154

Earlier quoted context omitted.

>The project you cite is super-optimized for the Prometheus use-case and data model. That may be true, but: Instead of figuring out how to meet the user's needs you're going to say the user is wrong?

That's not what he said and is kinda rude to be honest. What I consider to be said is that they optimize for extendability ( not just one use-case) and that those 2000 lines do a lot of things. Including replicating optimizations that are are not easy to achieve out of the box in their and other solutions. But they mostly found a way ;)

It's about as rude as telling someone their criticism is misplaced in that way. You've added more words to why you believe that, but that doesn't change the original sentiment.

You can say you don't want to support that usecase but someone inquired about whether you would. You said no, they're wrong to compare the two. Sounds a lot like "you're holding it wrong."

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#293
post #221

Earlier quoted context omitted.

Zabbix has been around for quite a long time. Easily 15 years now. I haven't looked at it since around 2013, but at the time it was placing quite some pressure on a mysql db backend. It looks like they've expanded out to support more than MySQL as the back-end these days.

Zabbix has been growing A LOT lately, and in a good way. It's nice to see this king of projects evolving Una good direction instead of stagnating and the diying.

It's great to hear. I liked zabbix in general. At the time of its initial surge in popularity, nginx and cacti were the dominant force in monitoring. Zabbix was a little quirky to get used to, but a breath of fresh air.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#294
post #3

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

Is that really true? Do you have any reference for that, I’m curious how different it is. FWIW I use InfluxDB and haven’t really thought it was a huge resource hog.

InfluxDB may require big amounts of RAM when working with high cardinality data. It works OK with up to a million of time series. Then it starts eating RAM as crazy [1]. How many time series does your InfluxDB setup contain?

[1] https://medium.com/@valyala/insert-benchmarks-with-inch-infl... .

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#295
post #40

Earlier quoted context omitted.

I guess the https://github.com/prometheus/pushgateway could help with that? As for the query performance there's a lot of things you can do with recording rules, that might help a lot with speeding up dashboards or queries.

Yeah the pushgateway was the alternative to using InfluxDB. In the end we actually used Datadog for it, despite the cost, as it was just easier to scale on it (we had hundreds of devices per site). The pushgateway route with Prometheus just ended up feeling like there were too many things relying on each other, i.e. Prometheus -> Push Gateway <- Multiple agents on each device, is inherently more complex than just con…

Try VictoriaMetrics next time. It supports data push via multiple popular data ingestion protocols [1] and it provides Prometheus-compatible API for Grafana [2].

[1] https://victoriametrics.github.io/#how-to-import-time-series...

[2] https://victoriametrics.github.io/#grafana-setup

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#296
post #28

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

I don't. It's quite difficult to do complex queries and its query language has a few gotcha. I think it's quite good and one of the best solutions today, but I look forward to something as simple and as fast, but with a proper query language.

PromQL from Prometheus has steep learning curve because of lack of good documentation. When you understand PromQL basics, then it is much easier to write typical queries over time series data in PromQL than in any other query languages (SQL, Flux, InfluxQL, etc.). I'd recommend reading the article about PromQL basics [1] and then feel the power of PromQL.

[1] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#297

We switched from InfluxDB to TimescaleDB for our IoT solutions. InfluxDB is very difficult to work with large datasets and enterprise/region compliance. We ingest around 100MB data per day and growing.

100MB data per day looks like a tiny number. Our users ingest 100GB data per day at the ingestion rate of 1M data points per second on a single-node setup [1].

[1] https://victoriametrics.github.io/CaseStudies.html

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#298
post #96

I've only ever used third party monitoring tools, but hope to set up a startup again soon and want to do OSS if I can. Can anyone comment on Prometheus vs Timescale? What are the tradeoffs? Or would I use Prometheus on top of Timescale?

Prometheus is a monitoring system (it collects metrics and evaluates alerts on them), while TimescaleDB is a database. While it is possible to use TimescaleDB as a remote storage for Prometheus, I'd recommend taking a look at other remote storage integrations as well [1]. Some of them natively support PromQL (like VictoriaMetrics, m3db, Cortex), others may be easier to setup and operate. For instance, VictoriaMetrics works out of the box without complex configuration. Disclaimer: I work for VictoriaMetrics :).

[1] https://prometheus.io/docs/operating/integrations/#remote-en...

Post reply on HN