Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

Post by user5994461 » Tue, Jul 21, 2020, 4:31 PM UTC

ciguy wrote: post #59

carlosdp wrote: post #46

Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.

This is good advice, however you will also want to make sure you have a plan to get off datadog when you grow. Datadog is one of the easiest to use and most comprehensive out of the box. But it gets really expensive as you begin to scale up and add servers, cloud accounts, services etc.... At a certain scale, rolling your own monitoring and alerting becomes cost effective again as Datadog begins to charge an arm and…

Post by gshulegaard » Tue, Jul 21, 2020, 4:35 PM UTC

avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

Post by user5994461 » Tue, Jul 21, 2020, 4:36 PM UTC

dijit wrote: post #148

user5994461 wrote: post #127

Earlier quoted context omitted.

Western salaries are not that low and true costs to the employer is typically double the perceived salary. That means we're talking tens of thousands of euros, so thousands of hosts (list price is certainly negotiable at this scale). If they've got a thousand of hosts, the costs of the infrastructure itself must dwarf the salary of any developer by orders of magnitude, the salary of a developer is simply irrelevant w…

> true costs to the employer is typically double the perceived salary. This is true, but my comment was an offhanded way to say that my "salary" (as in, the one on my contracts and the one I "see") is less than a month of Datadog for our number of hosts. As for the rest of your comment, I wish it was true. Developer salaries outside of the capitals is quite low in Europe, and even inside the capitals only go to "near…

Post by k-rus » Tue, Jul 21, 2020, 4:41 PM UTC

Twirrim wrote: post #144

hnarn wrote: post #129

Earlier quoted context omitted.

I haven't, unfortunately, but it looks promising from just looking into it briefly. Open source monitoring is always an area that needs more competition.

Zabbix has been around for quite a long time. Easily 15 years now. I haven't looked at it since around 2013, but at the time it was placing quite some pressure on a mysql db backend. It looks like they've expanded out to support more than MySQL as the back-end these days.

Post by dharmab » Tue, Jul 21, 2020, 4:49 PM UTC

gen220 wrote: post #93

lima wrote: post #65

Earlier quoted context omitted.

Please elaborate - how is it one engineer's full time job? We run Prometheus in production and this hasn't been our experience at all. A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).

I think the parent's complaint is a function of how your engineering org "uses" prometheus. If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus). IMO, this is a misuse of the technology, but…