Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

Post reply on HN
Print view View original

181–190 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#181

Post by mhall119 »

rattray wrote: post #177
mhall119 wrote: post #130

Earlier quoted context omitted.

Why is that? It takes minimal effort to get it up and running and you can either self-host or use the SaaS offering on any of the major clouds. There's even a free tier on the SaaS your startup can use that won't cost you a dime until your usage becomes significant.

Because the VPE sounds like the corporate sort of person who would not prioritize the things I'd want them to as a customer.

I don't think anybody who knows Ryan would describe him as "the corporate sort of person" :)

He was giving you an honest assessment of what was going on, not sugar-coated or wrapped in corporate speak. The work he described on our SaaS offering was directly tied to what our current and potential customers wanted.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#182

Post by mhall119 »

pachico wrote: post #172
mhall119 wrote: post #95

Earlier quoted context omitted.

> From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising. Don't read too much into that, it's more a result of wanting to get testable releases out early and temporarily redirecting engineering resources to take advantage of opportunities that arose in that timeframe than anything having to do with the code of 2.0 itself. Our Cloud 2 SaaS offe…

Fair enough. Good luck then with 2.0. I just wish the value proposition is strong enough to inspire users like me to try it again although much has to change for that to happen. I just wish I didn't have the feeling that the free version is just a bait for the SaaS. Anyhow, as said, good luck.

The OSS version is going to fill a vital role that SaaS just can't do, and we view it as a critical component in what InfluxDB provides. I don't think we're ever going to get away from needing on-prem and on-device deployments when dealing with time-series data, especially for emerging IoT/Edge use cases.

No, the company recognizes that the success of the SaaS is inevitably tied to the success of the OSS product. The only reason we were able to give the SaaS more focus recently is because we already had a working OSS product in the 1.x line that was meeting the needs of existing customers that we were and are continuing to invest in.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#183

Post by themgt »

avthar wrote: post #82
jimaek wrote: post #81

Earlier quoted context omitted.

While influx is pretty bad overall it's super simple to deploy and configure unlike timescale. It's the main reason we decided to use influx in our small team with simple enough timeseries needs

Curious what you found difficult to deploy/ configure? Is this in a self-managed context?

I was testing out both TimescaleDB/InfluxDB recently, including maybe using with Prometheus/Grafana. I was leaning towards Timescale, but InfluxDB was indeed a lot easier to quickly boot a "batteries included" setup and start working with live data.

I eventually spent a while reading about Timescale 1 vs 2, and testing the pg_prometheus[1] adapter and started thinking through integrating its schema to our other needs then realizing it's "sunsetted" and then reading about the new timescale-prometheus[2] adapter and reading through its ongoing design doc[3] with updated schema that I'm less a fan of.

I finally wound up mostly-settling on Timescale although I've put the Prometheus extension question on hold, just pulling in metrics data and outputting with ChartJS and some basic queries got me a lot closer to done for now. Our use case may be a little odd regardless, but I think a timescale-prometheus extension with a some ability to customize how the data is persisted would be quite useful.

[1] https://github.com/timescale/pg_prometheus

[2] https://github.com/timescale/timescale-prometheus

[3] https://docs.google.com/document/d/1e3mAN3eHUpQ2JHDvnmkmn_9r...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#184

Post by etxm »

avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

> high-cardinality workloads

Does this mean if you were using it with Prometheus you could get around issues with high cardinality labels?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#185

Post by ggregoire »

second--shift wrote: post #24

I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this? I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The mon…

I think you are overestimating the time and complexity to install and set up prometheus + grafana on a box, node exporter on your hosts and copy/paste a grafana dashboard for node exporter (which is your use case).

It gets complex only when you start monitoring your apps (i.e. using a prometheus client library to generate and export custom app metrics) and create custom grafana dashboards for these metrics. Or if you need to monitor some niche technology without its own existing prometheus exporter. Then yes, you need to read the docs, think about what you need to monitor and how, write code...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#186

Post by yellow_mixer »

Axsuul wrote: post #66
carlosdp wrote: post #46

Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.

As a counterpoint: I have been rolling my own Grafana/Prometheus for many years for my startup and it's been pretty trivial

How are you running it? On AWS? What’s your infra look like.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#187

Post by dijit »

user5994461 wrote: post #176
dijit wrote: post #152

Earlier quoted context omitted.

It's not millions, but it's the many multiples of hundreds of thousands. (and it's mainly GCP/bare metal). I guess the point I am driving at here is that there's such a thing as "business critical costs" (IE: can we ship our product or not) which is the majority of infra costs we have today, and then there's "optimisation costs". Usually when we discuss things like optimisation costs its along the lines of: "Will thi…

Great choice on GCP! It's probably half of the price as AWS for the same thing. Monitoring and logging are business critical. It's an integral part of infrastructure and it is very normal to spend 10% there. It's really not possible to operate stably and efficiently at a large scale like that without a trove of tooling. Tools usually justify their costs by allowing to optimize the infra and helping to prevent/fix out…

The community versions of ELK, Zabbix, InfluxDB and grafana.

Zabbix is the weak link here for sure, but the monitoring is quite comprehensive.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#188

Post by Axsuul »

yellow_mixer wrote: post #186
Axsuul wrote: post #66

Earlier quoted context omitted.

As a counterpoint: I have been rolling my own Grafana/Prometheus for many years for my startup and it's been pretty trivial

How are you running it? On AWS? What’s your infra look like.

GCP with Docker Swarm, scheduled on the same node (n2-standard-4). Let me know if any other questions

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#189

Post by dfinninger »

halfmatthalfcat wrote: post #33
x87678r wrote: post #28

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

How about another Prometheus server in federation[0]?

It's as simple as another Prom instance pointing at the "live" Prom instance. You can filter only the metrics you want to keep for a long amount of time, and downsample if necessary (by just setting a higher scrape interval on the "long term" Prom).

Since this "long term" Prom isn't in the critical path you could skimp on processing resources and just give it a big disk as a cost optimization.

If you are on AWS (or equivalent) the storage there is pretty durable. On-prem you can run HA (two instances with identical config).

[0] https://prometheus.io/docs/prometheus/latest/federation/

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#190

Post by pachico »

mhall119 wrote: post #182
pachico wrote: post #172

Earlier quoted context omitted.

Fair enough. Good luck then with 2.0. I just wish the value proposition is strong enough to inspire users like me to try it again although much has to change for that to happen. I just wish I didn't have the feeling that the free version is just a bait for the SaaS. Anyhow, as said, good luck.

The OSS version is going to fill a vital role that SaaS just can't do, and we view it as a critical component in what InfluxDB provides. I don't think we're ever going to get away from needing on-prem and on-device deployments when dealing with time-series data, especially for emerging IoT/Edge use cases. No, the company recognizes that the success of the SaaS is inevitably tied to the success of the OSS product. The…

I must say that if scalability is not part of OSS then it inevitably smells fishy to me. I am willing to pay to avoid the pain of maintaining or designing clusters but not because it's the only way to scale up. There are too many options that allow to scale with OSS version now it's hard to justify it. It's still a no-go for me.

Again, just take it as one person's opinion. I can't even grasp how complicated and challenging is to run what you offer. This is strictly user's opinion.

Post reply on HN