Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

91–100 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#91
post #68

Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.

I just costed a datadog deployment (based on your comment) and it would cost me my yearly salary every month. No thanks. :/

Can you say something vague about your deployment scale?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#92
post #68

Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.

I just costed a datadog deployment (based on your comment) and it would cost me my yearly salary every month. No thanks. :/

Wait, you're saying it costs 12x as much as your salary? That doesn't seem right... the company I'm at uses it pretty heavily and we're at about 1x of a FTE salary (and it's still super worth it)

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#93
post #65
post #60

Earlier quoted context omitted.

Prometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.

Please elaborate - how is it one engineer's full time job? We run Prometheus in production and this hasn't been our experience at all. A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).

I think the parent's complaint is a function of how your engineering org "uses" prometheus.

If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus).

IMO, this is a misuse of the technology, but a lot of unicorn startups have invested a lot of engineering resources into using it this way. And a lot of new companies are using it this way; hence the "one engineer's FT job".

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#94
post #90
post #86

Earlier quoted context omitted.

I don't consider TimescaleDB to be a serious contender as long as I need a 2000 line script to install functions and views to have something essential for time-series data like dimensions: https://github.com/timescale/timescale-prometheus/blob/maste... https://github.com/timescale/timescale-prometheus/blob/maste...

(Not affiliated with TimescaleDB, just trying to understand your critique) TimescaleDB is a Postgres extension, which means they have to play PG's rules, which rather implies a complicated mesh of functions, views, and "internal-use-only" tables. But once they're there, you can pretty much pretend they don't exist (until they break, of course, but this is true for everything in your software stack). Is your complaint…

Which kinda confirms the parent’s issues... InfluxDB is really easy to deploy and forget. With TimescaleDB you should be ready to know ins-and-outs of PG to secure and maintain correctly. Sure, for scaling and high loads TDB might be good but InfluxDB is easier and suitable for most loads and maintainability.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#95
post #22

I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome. I'm now having better results and experience storing data in ClickHouse…

> From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising.

Don't read too much into that, it's more a result of wanting to get testable releases out early and temporarily redirecting engineering resources to take advantage of opportunities that arose in that timeframe than anything having to do with the code of 2.0 itself.

Our Cloud 2 SaaS offering is already running the 2.0 code in production (albeit with changes to support it being deployed as a massively multi-tenant service)

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#97
post #68

Earlier quoted context omitted.

I just costed a datadog deployment (based on your comment) and it would cost me my yearly salary every month. No thanks. :/

Wait, you're saying it costs 12x as much as your salary? That doesn't seem right... the company I'm at uses it pretty heavily and we're at about 1x of a FTE salary (and it's still super worth it)

World’s salaries aren’t only US based :) Datadog for a smaller Western Europe startup is going to cost more than their devs salaries.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#98
post #8

I'm curious why they chose CollectD and not Prometheus or another pull-based ("scraping") solution. Pulling is more compositional than pushing and requires fewer machines to be touched when monitoring configuration is changed.

The author responded in another thread, it was just due to being familiar with CollectD already

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#100
As mentioned by others I have found a Prometheus + Grafana setup to be pretty straight forward to install and maintain. It also aligns with the cloud native foundation which means that this stack easily integrates with a bunch of OSS stack used by cloud native companies.
Post reply on HN