Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

Post reply on HN
Print view View original

171–180 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#171

Post by rckoepke »

crecker wrote: post #57
gjulianm wrote: post #47

Earlier quoted context omitted.

I'm not experienced with the CollectD stack, but I use Prometheus + Grafana to monitor probes. My two cents: - Fairly lightweight. Prometheus deals with quite a lot of series without much memory or CPU usage. - Integration with a lot of applications. Prometheus lets me monitor not only the system, but other applications such as Elastic, Nginx, PostgreSQL, network drivers... Sometimes I need an extra exporter, but the…

Author here. I'll probably write another tutorial focusing on Prometheus, instead of CollectD. Thanks for suggestion SerHack

It would be wonderful if you included limitations as well, to help people make the right decisions for their tech stack. I've been playing around with Prometheus lately for environmental monitoring, and long-term retention is particularly important to me.

During proof-of-concept testing, some historical data on disk perhaps wasn't lost per se, but definitely failed to load on restart. I haven't worked hard to replicate this but there are some similar unsolved tickets out there.

Additional traps for new players include customizing --storage.tsdb.retention.time and related parameters.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#172

Post by pachico »

mhall119 wrote: post #95
pachico wrote: post #22

I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome. I'm now having better results and experience storing data in ClickHouse…

> From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising. Don't read too much into that, it's more a result of wanting to get testable releases out early and temporarily redirecting engineering resources to take advantage of opportunities that arose in that timeframe than anything having to do with the code of 2.0 itself. Our Cloud 2 SaaS offe…

Fair enough. Good luck then with 2.0. I just wish the value proposition is strong enough to inspire users like me to try it again although much has to change for that to happen. I just wish I didn't have the feeling that the free version is just a bait for the SaaS. Anyhow, as said, good luck.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#173

Post by lma21 »

victor106 wrote: post #20
wiradikusuma wrote: post #13

There's also https://github.com/timberio/vector

+1 for vector. We moved from Logstash to Vector and we couldn't be happier. Logstash is awesome but its a memory hog. With Vector and Toshi you can kinda (I am not sure Toshi is as mature as Elastic) use them to replace LogStash and Elastic, the missing piece is Kibana

Does Vector work with TimescaleDB? it looks quite interesting

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#174

Post by Legogris »

gen220 wrote: post #106
klohto wrote: post #94

Earlier quoted context omitted.

Which kinda confirms the parent’s issues... InfluxDB is really easy to deploy and forget. With TimescaleDB you should be ready to know ins-and-outs of PG to secure and maintain correctly. Sure, for scaling and high loads TDB might be good but InfluxDB is easier and suitable for most loads and maintainability.

> With TimescaleDB you should be ready to know ins-and-outs of PG to secure and maintain correctly. Gotcha, this makes sense. To this, I'd pose the question: is the same not true for Influx? (i.e. With IFDB, you should be ready to know its ins and outs to secure and maintain it correctly). I guess I think about choosing a database like buying a house. I want it to be as good in X years as it is today, maybe better. F…

Conversely, because of its larger scope and longer history, there are a lot more ins and outs for PG.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#175

Post by heliodor »

jimaek wrote: post #81
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

While influx is pretty bad overall it's super simple to deploy and configure unlike timescale. It's the main reason we decided to use influx in our small team with simple enough timeseries needs

For small and medium needs, InfluxDB is a delight to use. On top of that, the data exploration tool within Chronograf is one-of-a-kind when it comes to exploring your metrics.

If you are looking for a vendor to host, manage, and provide attentive engineering support, check out my company: https://HostedMetrics.com

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#176

Post by user5994461 »

dijit wrote: post #152
user5994461 wrote: post #143

Earlier quoted context omitted.

2500 instances could be millions per months in AWS costs. The smallest instances with some disks and bandwidth fees can push 100k a month. Spending a fraction of that to monitor that sort of infrastructure is absolutely justified. I can tell you from experience that datadog gives discount even for 100+ hosts, I don't know what they can do for 2500, but if it were me I wouldn't accept anything less than 50% off. Hones…

It's not millions, but it's the many multiples of hundreds of thousands. (and it's mainly GCP/bare metal). I guess the point I am driving at here is that there's such a thing as "business critical costs" (IE: can we ship our product or not) which is the majority of infra costs we have today, and then there's "optimisation costs". Usually when we discuss things like optimisation costs its along the lines of: "Will thi…

Great choice on GCP! It's probably half of the price as AWS for the same thing.

Monitoring and logging are business critical. It's an integral part of infrastructure and it is very normal to spend 10% there. It's really not possible to operate stably and efficiently at a large scale like that without a trove of tooling.

Tools usually justify their costs by allowing to optimize the infra and helping to prevent/fix outages, though not all companies care about stability or hardware costs.

And it's not a choice of free vs paid. open source software costs a lot of money too, pairs of large instances to run it don't come cheap, they're probably more than a salary too if the company wants to have any sort of redundancy or geographic distribution.

May I ask what do you have for logging? I guess you must be screaming in horror at the price of elasticsearch/kibana/splunk :D

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#177

Post by rattray »

mhall119 wrote: post #130
rattray wrote: post #103

Earlier quoted context omitted.

Well, guess I know to never use Influx as a startup

Why is that? It takes minimal effort to get it up and running and you can either self-host or use the SaaS offering on any of the major clouds. There's even a free tier on the SaaS your startup can use that won't cost you a dime until your usage becomes significant.

Because the VPE sounds like the corporate sort of person who would not prioritize the things I'd want them to as a customer.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#178

Post by rattray »

k-rus wrote: post #117
rattray wrote: post #96

I've only ever used third party monitoring tools, but hope to set up a startup again soon and want to do OSS if I can. Can anyone comment on Prometheus vs Timescale? What are the tradeoffs? Or would I use Prometheus on top of Timescale?

You can use Prometheus on top of TimescaleDB. Timescale builds connector and entire workflow to run Prometheus on top of TimescaleDB and support Grafana in flexible way. Sorry for the promo :) check for details in https://github.com/timescale/timescale-prometheus

Thanks!

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#180

Post by user5994461 »

lima wrote: post #65
dharmab wrote: post #60

Earlier quoted context omitted.

Prometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.

Please elaborate - how is it one engineer's full time job? We run Prometheus in production and this hasn't been our experience at all. A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).

For reference: A single machine can push 5k metrics so you're saying a single prometheus instance can easily serve 20s of hosts. lol
Post reply on HN