Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.
I just costed a datadog deployment (based on your comment) and it would cost me my yearly salary every month. No thanks. :/
Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
91–100 of 298 posts
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#92Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.
I just costed a datadog deployment (based on your comment) and it would cost me my yearly salary every month. No thanks. :/
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#93Earlier quoted context omitted.
Prometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.
Please elaborate - how is it one engineer's full time job? We run Prometheus in production and this hasn't been our experience at all. A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).
If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus).
IMO, this is a misuse of the technology, but a lot of unicorn startups have invested a lot of engineering resources into using it this way. And a lot of new companies are using it this way; hence the "one engineer's FT job".
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#94Earlier quoted context omitted.
I don't consider TimescaleDB to be a serious contender as long as I need a 2000 line script to install functions and views to have something essential for time-series data like dimensions: https://github.com/timescale/timescale-prometheus/blob/maste... https://github.com/timescale/timescale-prometheus/blob/maste...
(Not affiliated with TimescaleDB, just trying to understand your critique) TimescaleDB is a Postgres extension, which means they have to play PG's rules, which rather implies a complicated mesh of functions, views, and "internal-use-only" tables. But once they're there, you can pretty much pretend they don't exist (until they break, of course, but this is true for everything in your software stack). Is your complaint…
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#95I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome. I'm now having better results and experience storing data in ClickHouse…
Don't read too much into that, it's more a result of wanting to get testable releases out early and temporarily redirecting engineering resources to take advantage of opportunities that arose in that timeframe than anything having to do with the code of 2.0 itself.
Our Cloud 2 SaaS offering is already running the 2.0 code in production (albeit with changes to support it being deployed as a massively multi-tenant service)
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#96Can anyone comment on Prometheus vs Timescale? What are the tradeoffs? Or would I use Prometheus on top of Timescale?
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#97Earlier quoted context omitted.
I just costed a datadog deployment (based on your comment) and it would cost me my yearly salary every month. No thanks. :/
Wait, you're saying it costs 12x as much as your salary? That doesn't seem right... the company I'm at uses it pretty heavily and we're at about 1x of a FTE salary (and it's still super worth it)
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#98I'm curious why they chose CollectD and not Prometheus or another pull-based ("scraping") solution. Pulling is more compositional than pushing and requires fewer machines to be touched when monitoring configuration is changed.
Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
#99Too many parts Netdata will make it a breeze