Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

Post reply on HN
Print view View original

231–240 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#231

Post by didip »

k-rus wrote: post #211
didip wrote: post #196

Earlier quoted context omitted.

How does multi node works with Postgres? Does TimescaleDB create its own Raft layer on top and just treat Postgres as dumb storage?

A TimescaleDB engineer here. Current implementation of database distribution in TimescaleDB is centralised where all traffics go through an access node, which distributes the load into data nodes. The implementation uses 2PC. Abilities of PostgreSQL to generate distributed query plans are utilised together with TimescaleDB optimisations. So PostgreSQL is used not just a dumb storage :)

Won't that access node be a single point of failure then? Just trying to learn more.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#232

Post by threeseed »

rattray wrote: post #177
mhall119 wrote: post #130

Earlier quoted context omitted.

Why is that? It takes minimal effort to get it up and running and you can either self-host or use the SaaS offering on any of the major clouds. There's even a free tier on the SaaS your startup can use that won't cost you a dime until your usage becomes significant.

Because the VPE sounds like the corporate sort of person who would not prioritize the things I'd want them to as a customer.

Your issue is that they are prioritizing paying customers over giving you a free product.

I don't think that's fair given they are still a startup.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#233

Post by drez »

linsomniac wrote: post #111

I've been doing monitoring of our ~120ish machines for 3-4 years now using Influx+Telegraf+Grafana, and have been really happy with it. Prior to that we were using collectd+graphite and with 1 minute stats it was adding some double-digits %age utilization on our infrastructure (I don't remember exactly how much, but I want to say 30% CPU+disk). Influxdb has been a real workhorse. We suffered through some of their ear…

I just wanted to echo this sentiment. Influx had its fair share of issues a few years ago (v0.8 migration, changing storage engines, tag cardinality issues), but the latest v1.x releases have been solid. I have been using the TIK stack (I use Grafana instead of Chronograf) for monitoring several dozen production-facing machines for 2 years now without a single issue, which I would very much count as a win.

I just hope they learned their lesson for the v2.0 release...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#234

Post by ngrilly »

avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

TimescaleDB and my team is using it, but one significant drawback compared to solutions like Prometheus are the limitations of continuous aggregations (basically no joins, no order by, no window functions). That’s a problem when you want to consolidate old data.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#235

Post by deepakhj »

carlosdp wrote: post #206
jakozaur wrote: post #115

Earlier quoted context omitted.

DataDog per host pricing can be very expensive. Metrics are provided by many platforms. If you need logs too, you may look at Sumo Logic which got way cheaper metrics in typical use case. Disclaimer: I work at Sumo Logic.

It's expensive for good reason, it's the best out there.

That's debatable. I prefer Grafana.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#236

Post by deepakhj »

halfmatthalfcat wrote: post #33
x87678r wrote: post #28

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

Thanos or Cortex

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#237

Post by yellow_mixer »

Axsuul wrote: post #213
carlosdp wrote: post #210

Earlier quoted context omitted.

Does "scheduled on the same node" imply you are running everything on one node?

Nope, just that Prometheus and Grafana are running together on the same node. My entire infrastructure is multi-node however.

How much data are you sending and ingesting?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#238

Post by cevian »

ngrilly wrote: post #234
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

TimescaleDB and my team is using it, but one significant drawback compared to solutions like Prometheus are the limitations of continuous aggregations (basically no joins, no order by, no window functions). That’s a problem when you want to consolidate old data.

(TimescaleDB engineer) we hear you and are working on making continuous aggregations easier to use.

For now, the recommended approach is to perform continuous_aggregates on single tables and perform joins, order by, and window when querying the materialized aggregate rather than when materializing.

This often has the added benefit of often making the materialization more general so that a wider range of queries can use it.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#239

Post by cevian »

didip wrote: post #231
k-rus wrote: post #211

Earlier quoted context omitted.

A TimescaleDB engineer here. Current implementation of database distribution in TimescaleDB is centralised where all traffics go through an access node, which distributes the load into data nodes. The implementation uses 2PC. Abilities of PostgreSQL to generate distributed query plans are utilised together with TimescaleDB optimisations. So PostgreSQL is used not just a dumb storage :)

Won't that access node be a single point of failure then? Just trying to learn more.

The access node is replicated using streaming replication and is thus not a SPOF.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#240

Post by hagen1778 »

codeduck wrote: post #194
halfmatthalfcat wrote: post #33

Earlier quoted context omitted.

The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

For long term retention, look at VictoriaMetrics

My vote is for VictoriaMetrics - the best option if you want to keep things simple and get great performance in the same time!
Post reply on HN