Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

71–80 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#71
post #3

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

What about storage? We are running influxdb and we are looking for alternative. But a point where Influx is good is storage.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#72
post #31

Earlier quoted context omitted.

Classic "two masters" problem.

This was from the CTO last week: "This work won't be landing anywhere until sometime next year and it'll be landing in our Cloud 2 offering first." So the OSS is definitely a second-class citizen. And now that they've dropped all DevRel activity, don't expect much attention for OSS Developers and users.

Quite the contrary, we're finishing up packaging of the OSS version now to make it as easy as possible for you to get it and we're assembled a new team focused on getting OSS 2.0 released.

We continue to support all users, including OSS users, in our public Slack and Discourse, as well as Github. We have not "dropped all DevRel" activity.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#73
post #28

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

I don't. It's quite difficult to do complex queries and its query language has a few gotcha. I think it's quite good and one of the best solutions today, but I look forward to something as simple and as fast, but with a proper query language.

Yes would be nice to be able to run sql or something more standard. I like that its just simple, one application and it just works.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#74
[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention of PCAPs) and save some security logs (I'm not intending on alerting based on logs, just for retention). But being able to have a network overview, insight into current connections (including history) is a very useful thing. Can anybody recommend something, that's maybe a bit lighter than an entire Elastic-Stack?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#75
post #60
post #52

Earlier quoted context omitted.

I'd be interested to know a bit of detail as we're looking into Grafana/Promethus a bit.

Prometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.

False in my experience. Full-time job? After the initial learning curve, a simple 2x redundant Prometheus poller setup on can last for a long time. Ours lasted for 30,000,000 timeseries until encountering performance issues.

After that, we needed some more effort to scale out horizontally with Thanos, but again, once it's set up, it maintains itself.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#77

There's also https://github.com/timberio/vector

+1 for vector. We moved from Logstash to Vector and we couldn't be happier. Logstash is awesome but its a memory hog. With Vector and Toshi you can kinda (I am not sure Toshi is as mature as Elastic) use them to replace LogStash and Elastic, the missing piece is Kibana

Have you looked into Grafana Loki for logs? If I had to redo one part of our stack, that's what I'd choose.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#78
post #3

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

InfluxDB is pretty good if you don't need to do any advanced querying like grouping by month or formulas. Its strong points are low diskspace footprint and very fast queries even over long periods of time.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#79
post #71
post #3

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

What about storage? We are running influxdb and we are looking for alternative. But a point where Influx is good is storage.

You mean storage efficiency? Seems that unless the aggregation is happening before being stored, it would be unfair to compare influxdb with other databases that are tasked with storing per-record granularity.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#80
post #28

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

I second the Thanos link, but if you want to only use Prometheus, I would look into federation:

https://www.robustperception.io/scaling-and-federating-prome...

https://prometheus.io/docs/prometheus/latest/federation/

You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.

Post reply on HN