Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD • Hacker News Board

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#31

Post by edoceo » Tue, Jul 21, 2020, 1:57 PM UTC

rbetts wrote: post #23

candiddevmike wrote: post #7

Earlier quoted context omitted.

Influx is really shitting the bed with how they're handling the InfluxDB 2.0 release. The docs are a mess, and the migration tool seems like the result of a weekend hackathon. They're leaving a lot of customers with long term metrics in a tough spot. If you're thinking about using Influx for long term data storage, look elsewhere. The company continuously burns customer goodwill by going against the grain, and buckin…

(I'm the VPE at Influxdata). I appreciate this sentiment. We've been focused on a building a SaaS version of Influxdata and are committed to a paired open source version of that. The open source version has been lagging as we work on the SaaS side. However, we are committed to shipping a GA version of the OSS 2.0 stack around the end of Q3 that offers an in-place data migration capability from 1.x OSS. We've spoken a…

Classic "two masters" problem.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#32

Post by robotmay » Tue, Jul 21, 2020, 2:06 PM UTC

ddevault wrote: post #26

I found the software in this stack to be very bloated and difficult to maintain. Large, complicated software has a tendency to fall flat on its face when something goes wrong, and this is a domain where reliability is paramount. I wrote about a different approach (Prometheus + Alertmanager) for sourcehut, if you're curious: https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...

I have a bit of a love/hate relationship with Prometheus. At home I really like it; it was simple to set up for my needs and most of my configuration is on my server which then scrapes other machines for the data. However I find it quite frustrating at scale for work, both in its concepts (it's hard to describe but it's sort of...backwards?) and in its query performance, although that might be a side-effect of using it with Grafana and me attempting to misuse it. By contrast I think the concepts of something like TimescaleDB are easier to understand when it comes to scaling and optimising that service.

In my previous job I had a very clear use-case for not using Prometheus and did for a while use InfluxDB (it involved devices sending data from behind firewalls across many sites). I found it pretty expensive to scale and it fell over when it ran out of storage, which feels like something that should have been handled automatically considering it was a PaaS offering.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#33

Post by halfmatthalfcat » Tue, Jul 21, 2020, 2:07 PM UTC

x87678r wrote: post #28

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

The question is how to do long term storage though. Something I've had a bit of trouble rationing about.

Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#34

Post by majkinetor » Tue, Jul 21, 2020, 2:07 PM UTC

You can quickly do this on Windows:

    cinst influxdb1 /Service
    cinst grafana
    start $Env:ChocolateyInstall\lib\grafana\tools\grafana-*\bin\grafana-server.exe
    git clone https://github.com/majkinetor/psinflux
    import-module ./psinflux
    1..10 | % { $x = 10*$_ + (Get-Random 10); Send-Data "test1 value=$x"; sleep 1 }

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#35

Post by ddevault » Tue, Jul 21, 2020, 2:08 PM UTC

robotmay wrote: post #32

ddevault wrote: post #26

I found the software in this stack to be very bloated and difficult to maintain. Large, complicated software has a tendency to fall flat on its face when something goes wrong, and this is a domain where reliability is paramount. I wrote about a different approach (Prometheus + Alertmanager) for sourcehut, if you're curious: https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...

I have a bit of a love/hate relationship with Prometheus. At home I really like it; it was simple to set up for my needs and most of my configuration is on my server which then scrapes other machines for the data. However I find it quite frustrating at scale for work, both in its concepts (it's hard to describe but it's sort of...backwards?) and in its query performance, although that might be a side-effect of using…

One point of note for SourceHut's Prometheus use is that we generally don't make dashboards. I don't really like Grafana. I will sometimes use gnuplot with styx to plot graphs on an as-needed basis:

https://github.com/go-pluto/styx

This is how I made the plots in that blog post.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#36

Post by dewey » Tue, Jul 21, 2020, 2:08 PM UTC

second--shift wrote: post #24

I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this? I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The mon…

At work and in my personal projects I use a Prometheus + Grafana setup and it's very easy to set up. Not sure what you mean with the complex and setup costs for a Grafana stack?

Alerting with the Prometheus AlertManager is also pretty straight forward and I'm looking at dashboards every day to see if everything is running smoothly or tracking down what's not working well if there are any issues. Grafana dashboards are always the second thing I look at after an alert fires somewhere and it has been invaluable.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#37

Post by viraptor » Tue, Jul 21, 2020, 2:09 PM UTC

second--shift wrote: post #24

I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this? I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The mon…

What sar+rsync doesn't provide:

- app-specific metrics

- quick and easy way to build a number of graphs searching for correlation (how to slice the data to get results that explain issues)

- log/metrics correlation

- unified way to build alerts

- ad-hoc changes - while you're in the middle of an incident and want to get information that's just slightly different from existing, or filter out some values, or overlay a trend - how long would it take in your custom solution vs grafana?

And finally - grafana exists. Why would I write a custom graph generator from a custom data store if I can setup a collector + influx + grafana in a fraction of that time and get back more?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#38

Post by dewey » Tue, Jul 21, 2020, 2:09 PM UTC

halfmatthalfcat wrote: post #33

x87678r wrote: post #28

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

I think that's what https://thanos.io/ is for.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#39

Post by jimktrains2 » Tue, Jul 21, 2020, 2:12 PM UTC

viraptor wrote: post #25

jimktrains2 wrote: post #18

Earlier quoted context omitted.

> TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) is a known solution and a fairly standard way to add elements of monitoring to your system. Amusing that I've never heard of any of this but have heard and used collectd. It's obviously no where near as common as a lamp stack or anywhere near common at all, so asking why it over something else is answered by "someone made it up so it's better".

Collectd is only one part here. Did you need a solution for inline processing/aggregation and alerting? If not, you wouldn't run into TICK. It's not common overall, because few environments need to go that far. I don't get the comparison to LAMP popularity. Insects are more common than cars too. They're different things ¯\_(ツ)_/¯

I mentioned lamp as an example of an acronym and stack that would be more common to ask why it wasn't chosen. In retrospect that probably wasn't the ideal way of getting to my point that tick isn't a common stack so asking why not it because it's common seems weird.

> If not, you wouldn't run into TICK. It's not common overall, because few environments need to go that far.

That's my point though. The post I replied to acted as though everyone knows of and uses it, but provided no information on what makes it a better choice for such usecases.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#40

Post by dewey » Tue, Jul 21, 2020, 2:12 PM UTC

robotmay wrote: post #32

ddevault wrote: post #26

I found the software in this stack to be very bloated and difficult to maintain. Large, complicated software has a tendency to fall flat on its face when something goes wrong, and this is a domain where reliability is paramount. I wrote about a different approach (Prometheus + Alertmanager) for sourcehut, if you're curious: https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...

I have a bit of a love/hate relationship with Prometheus. At home I really like it; it was simple to set up for my needs and most of my configuration is on my server which then scrapes other machines for the data. However I find it quite frustrating at scale for work, both in its concepts (it's hard to describe but it's sort of...backwards?) and in its query performance, although that might be a side-effect of using…

I guess the https://github.com/prometheus/pushgateway could help with that? As for the query performance there's a lot of things you can do with recording rules, that might help a lot with speeding up dashboards or queries.