Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

Post reply on HN
Print view View original

131–140 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#131

Post by gen220 »

mhall119 wrote: post #114
gen220 wrote: post #106

Earlier quoted context omitted.

> With TimescaleDB you should be ready to know ins-and-outs of PG to secure and maintain correctly. Gotcha, this makes sense. To this, I'd pose the question: is the same not true for Influx? (i.e. With IFDB, you should be ready to know its ins and outs to secure and maintain it correctly). I guess I think about choosing a database like buying a house. I want it to be as good in X years as it is today, maybe better. F…

Think if InfluxDB is "move-in ready" when buying a house. Sure you won't know the electrical and plumbing as well as you will a "fixer-uppper" in the long run, but you'll have a place to sleep a lot faster and easier.

I think it's a good analogy; very reasonable people come to different conclusions on this problem IRL too.

I prefer older houses (at least pre-80s). They have problems that are very predictable and minor. If they had major problems, they would show them quite clearly in 2020.

My friends prefer newer properties, but this is an act of faith, that the builders today have built something that won't have severe structural/electrical/plumbing problems in 20 years.

Of course, if you only plan on living some place for 3-5 years, none of this matters; you want to live where you're comfortable.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#132

Post by pachico »

gregwebs wrote: post #87
pachico wrote: post #22

I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome. I'm now having better results and experience storing data in ClickHouse…

Have you tried Victoria Metrics? It originallyl started as a ClickHouse fork for time series but is now a re-write keeping some of the same principals.

Yes, we use it as long term retention for Prometheus. I was also tempted to use it as stand-alone dB but never got to it. Do you use it as such?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#133

Post by retzkek »

Papric0re wrote: post #74

[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention…

Maybe Loki [1] meets your needs? It lacks the analytics abilities of Elastic (e.g. what's the average response time [2]) but is much simpler to setup and use for jog aggregations, and has a pretty powerful query language for digging through and graphing log statistics (e.g. how many errors have been logged per hour). It's mainly being developed by Grafana Labs, so there's great integration in Grafana.

[1] https://grafana.com/oss/loki/

[2] I'd argue this sort of thing should be published as a metric anyways so you don't have to pull it out of the logs

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#134

Post by Papric0re »

floren wrote: post #84
Papric0re wrote: post #74

[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention…

I think Gravwell ( https://gravwell.io ) might be what you're looking for--but I work for Gravwell so I may be biased! If I can be forgiven a short sales pitch, we've built a format-agnostic storage & querying system that easily handles raw packets, Netflow records (v5, v9, and IPFIX), collectd data, Windows event logs, and more. You can see some screenshots at https://www.gravwell.io/technology We have a free tier w…

I don't mind you giving a small sales pitch...and maybe your product is indeed what I'm searching for. But your pricing model is instantly putting me off. Same as with Splunk, you end up with not being able to predict your cost and paying way too much. Tell me when you fix that and I might be interested ;)

Edit: Sorry, I was misreading your comment. Premium is unlimited...I will look into it, thanks. :)

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#135

Post by bsg75 »

aeyes wrote: post #86
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

I don't consider TimescaleDB to be a serious contender as long as I need a 2000 line script to install functions and views to have something essential for time-series data like dimensions: https://github.com/timescale/timescale-prometheus/blob/maste... https://github.com/timescale/timescale-prometheus/blob/maste...

There may be some extra install steps with Timescale, but that is balanced by having data in a platform accessible by a very well known API.

There are always tradeoffs, but in cases like mine, I find the basis of the platform - PostgreSQL - to provide a very useful set of features.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#136

Post by aeyes »

gen220 wrote: post #90
aeyes wrote: post #86

Earlier quoted context omitted.

I don't consider TimescaleDB to be a serious contender as long as I need a 2000 line script to install functions and views to have something essential for time-series data like dimensions: https://github.com/timescale/timescale-prometheus/blob/maste... https://github.com/timescale/timescale-prometheus/blob/maste...

(Not affiliated with TimescaleDB, just trying to understand your critique) TimescaleDB is a Postgres extension, which means they have to play PG's rules, which rather implies a complicated mesh of functions, views, and "internal-use-only" tables. But once they're there, you can pretty much pretend they don't exist (until they break, of course, but this is true for everything in your software stack). Is your complaint…

My problem is what these scripts do. When working with timeseries you want to be able to filter on dimensions/tags/labels.

TimescaleDB doesn't support this out of the box, the reason being Postgres' limitations on multi column indexes when you have data types like jsonb. What they have to do to work around this is to have a mapping table from dimensions/tags/labels which are stored in a jsonb field to integers which they then use to look the data up in the metrics table.

I would expect that a timeseries database is optimized for this type of lookup without hacks involving two tables.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#137

Post by cevian »

aeyes wrote: post #86
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

I don't consider TimescaleDB to be a serious contender as long as I need a 2000 line script to install functions and views to have something essential for time-series data like dimensions: https://github.com/timescale/timescale-prometheus/blob/maste... https://github.com/timescale/timescale-prometheus/blob/maste...

(TimescaleDB engineer here). This is a really unfair critique.

The project you cite is super-optimized for the Prometheus use-case and data model. TimescaleDB beats InfluxDB on performance even without these optimization. It's also not possible to optimize in this way in most other time-series databases.

These scripts also work hard to give users a UIUX experience that mimicks PromQL in a lot of ways. This isn't necessary for most projects and is a very specialized use-case. That takes up a lot of the 2000 lines you are talking about.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#138

Post by Papric0re »

retzkek wrote: post #133
Papric0re wrote: post #74

[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention…

Maybe Loki [1] meets your needs? It lacks the analytics abilities of Elastic (e.g. what's the average response time [2]) but is much simpler to setup and use for jog aggregations, and has a pretty powerful query language for digging through and graphing log statistics (e.g. how many errors have been logged per hour). It's mainly being developed by Grafana Labs, so there's great integration in Grafana. [1] https://gra…

Tanks for the tip. But I would still need some storage and Data Shipper, right? Or is Loki also taking care of storage?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#140

Post by mickeyben »

We're currently using InfluxDB but maintenance is something we'd like to stop doing on our monitoring stack.

Datadog is too expensive because of the number of hosts we have. So we're thinking of eventually going to a hosted InfluxDB setup.

But we also want to revisit other hosted solutions. Does someone have some experience with using Cloudwatch + Grafana? I've used Cloudwatch many years ago and it was clearly subpar to something like Influx. Is is better nowadays?

Post reply on HN