Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

Post reply on HN
Print view View original

261–270 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#261

Post by douglasheriot »

avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

I've been using InfluxDB, but not satisifed with limited InfluxQL, or over-complicated Flux query languages. I love Postgres so TimescaleDB looks awesome.

The main issue I've got is how to actually get data into TimescaleDB. We use telegraf right now, but the telegraf Postgres output pull request still hasn't been merged: https://github.com/influxdata/telegraf/pull/3428

Any progress on this?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#262

Post by akulkarni »

douglasheriot wrote: post #261
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

I've been using InfluxDB, but not satisifed with limited InfluxQL, or over-complicated Flux query languages. I love Postgres so TimescaleDB looks awesome. The main issue I've got is how to actually get data into TimescaleDB. We use telegraf right now, but the telegraf Postgres output pull request still hasn't been merged: https://github.com/influxdata/telegraf/pull/3428 Any progress on this?

There is a telegraf binary available here that connects to TimescaleDB: https://docs.timescale.com/latest/tutorials/telegraf-output-...

If you are looking to migrate data, then you might also want to explore this tool: https://www.outfluxdata.com/

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#263

Post by ciguy »

carlosdp wrote: post #205
ciguy wrote: post #59

Earlier quoted context omitted.

This is good advice, however you will also want to make sure you have a plan to get off datadog when you grow. Datadog is one of the easiest to use and most comprehensive out of the box. But it gets really expensive as you begin to scale up and add servers, cloud accounts, services etc.... At a certain scale, rolling your own monitoring and alerting becomes cost effective again as Datadog begins to charge an arm and…

On the contrary, the bigger you get, the more you need Datadog to scale. Once you grow big enough, you can make volume deals with them, you don't pay list price. I'd much rather pay them than have a single engineer have to spend any time managing infrastructure that isn't core to our product, especially since Datadog will always do it better.

I respectfully strongly disagree. At a certain scale when you can afford to have a full-time engineer working on it, open source tooling will give you a better, cheaper and more flexible solution since it's easy to customize. Datadog is good for the mainstream cases but it's not all that flexible for unusual or edge cases.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#264

Post by julioo »

julioo wrote: post #212
x87678r wrote: post #49

Earlier quoted context omitted.

Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.

In any case, Prometheus is throwing away data if a scrap can’t be done. As clearly described on the website, prometheus is not a metrics system. So influx and Prometheus are quite different.

Mistake in my previous message. I wanted to say that Prometheus is not a log system. Metrics could be lost in scraping issues. This is ok in some cases but you have to know that you can loose data.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#265

Post by Papric0re »

floren wrote: post #216
iudqnolq wrote: post #191

Earlier quoted context omitted.

When you say > Gravwell is developed and maintained by engineers expert in security and obsessed with high performance. Therefore our codebase is 100% proprietary and does not rely on open source software. We love open source, but we love our customers and their peace of mind a lot more! does that mean you've even rolled your own webserver? Programming language?

That's... not good copy. I think it must have been written long ago. We use open-source libraries (with compatible licenses, of course) and even maintain our own set of open-source code ( https://github.com/gravwell ). I'll talk to the guys who maintain the website and get that fixed. Thanks for pointing it out! Edit: We've had lots of people assume we use Elastic under the hood, so I wonder if that was just a (poorl…

Maybe you should just wipe that paragraph completely. I get that investors like to see that you are using proprietary code, but I wouldn't expect you to be faster with that. Especially when running against Elastic, which has over 1.400 contributors currently. But you don't necessarily need to. You can get me with being focused on the right thing and not bloating your software. Lot of big projects start to loose focus and start doing everything, hence become worse doing their main job.

Especially when it comes to security, I'd like to see the lowest complexity possible. Harden your software instead of feature-fu around. That would be a good USP (I've got the feeling that no vendor has realized this so far - but customers neither did).

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#266

Post by ngrilly »

ngrilly wrote: post #234
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

TimescaleDB and my team is using it, but one significant drawback compared to solutions like Prometheus are the limitations of continuous aggregations (basically no joins, no order by, no window functions). That’s a problem when you want to consolidate old data.

Meant "TimescaleDB is great and my team is using it"...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#267

Post by ngrilly »

cevian wrote: post #238
ngrilly wrote: post #234

Earlier quoted context omitted.

TimescaleDB and my team is using it, but one significant drawback compared to solutions like Prometheus are the limitations of continuous aggregations (basically no joins, no order by, no window functions). That’s a problem when you want to consolidate old data.

(TimescaleDB engineer) we hear you and are working on making continuous aggregations easier to use. For now, the recommended approach is to perform continuous_aggregates on single tables and perform joins, order by, and window when querying the materialized aggregate rather than when materializing. This often has the added benefit of often making the materialization more general so that a wider range of queries can u…

Thanks for the advice! Makes sense. We are doing something similar (aggregate on single table and join later). But still looking a solution to compute aggregated increments when there are counter resets.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#268

Post by scaryclam »

carlosdp wrote: post #46

Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.

This is not my experience at all. We've got it setup and monitoring all sorts of things. After the initial setup, the only reason we've needed to touch it is when we've introduced new things and wanted to update the config. In fact, data dog was far more of a pain, and far less useful than prometheus has been.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#269

Post by a10c »

avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

One of the biggest quirks that I had bumped up against with TimescaleDB is that it's backed by a relational database.

We are a company that ingests around 90M datapoints per minute across an engineering org of around 4,000 developers. How do we scale a timeseries solution that requires an upfront schema to be defined? What if a developer wants to add a new dimension to their metrics, would that require us to perform an online table migration? Does using JSONB as a field type allow for all the desirable properties that a first-class column would?

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#270

Post by rehevkor5 »

linsomniac wrote: post #111

I've been doing monitoring of our ~120ish machines for 3-4 years now using Influx+Telegraf+Grafana, and have been really happy with it. Prior to that we were using collectd+graphite and with 1 minute stats it was adding some double-digits %age utilization on our infrastructure (I don't remember exactly how much, but I want to say 30% CPU+disk). Influxdb has been a real workhorse. We suffered through some of their ear…

It's so much easier to write incorrect/misleading queries in influxql than in promql. And you can't perform operations between two different series names in influxdb, last I looked. That makes it impossible to do things like ratios or percentages unless you have control over your metrics, and structure them the way influx likes. Also, no support for calculating percentiles from Prometheus histogram buckets.
Post reply on HN