Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

Post reply on HN
Print view View original

281–290 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#281

Post by valyala »

a10c wrote: post #269
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

One of the biggest quirks that I had bumped up against with TimescaleDB is that it's backed by a relational database. We are a company that ingests around 90M datapoints per minute across an engineering org of around 4,000 developers. How do we scale a timeseries solution that requires an upfront schema to be defined? What if a developer wants to add a new dimension to their metrics, would that require us to perform…

90M datapoints per minute means 90M/60=1.5M datapoints per second. Such amounts of data may be easily handled by specialized time series databases even in a single-node setup [1].

> What if a developer wants to add a new dimension to their metrics, would that require us to perform an online table migration?

Specialized time series databases usually don't need defining any schema upfront - just ingest metrics with new dimensions (labels) whenever you wish. I'm unsure whether this works with TimescaleDB.

[1] https://victoriametrics.github.io/CaseStudies.html

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#282

Post by valyala »

douglasheriot wrote: post #261
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

I've been using InfluxDB, but not satisifed with limited InfluxQL, or over-complicated Flux query languages. I love Postgres so TimescaleDB looks awesome. The main issue I've got is how to actually get data into TimescaleDB. We use telegraf right now, but the telegraf Postgres output pull request still hasn't been merged: https://github.com/influxdata/telegraf/pull/3428 Any progress on this?

I believe PromQL [1] and MetricsQL [2] are much better suited for typical queries over time series data than SQL, Flux or InfluxQL.

[1] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

[2] https://victoriametrics.github.io/MetricsQL.html

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#283

Post by valyala »

jimaek wrote: post #81
avthar wrote: post #62

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today Timesc…

While influx is pretty bad overall it's super simple to deploy and configure unlike timescale. It's the main reason we decided to use influx in our small team with simple enough timeseries needs

If you want the best of both worlds, then try VictoriaMetrics. It is simple to deploy and operate and it is more resource-efficient comparing to InfluxDB. More on this, it supports data ingestion over Influx line protocol [1].

[1] https://victoriametrics.github.io/#how-to-send-data-from-inf...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#284

Post by user5994461 »

sciurus wrote: post #229
user5994461 wrote: post #143

Earlier quoted context omitted.

2500 instances could be millions per months in AWS costs. The smallest instances with some disks and bandwidth fees can push 100k a month. Spending a fraction of that to monitor that sort of infrastructure is absolutely justified. I can tell you from experience that datadog gives discount even for 100+ hosts, I don't know what they can do for 2500, but if it were me I wouldn't accept anything less than 50% off. Hones…

It could also be less than $100k a month (e.g. 2500 c5a.large with a 1-year reservation). At that point you'd wonder why your monitoring bill was 40% of your compute bill. Also, of course their salary is relevant. The cost of an engineer's time is an important factor to consider when making build vs buy decisions. Usually it's one that argues in favor of "buy", but not always.

2500 c5a.large is $172k per month, or $102k with one year reservation full upfront.

Add 10% support fees, EBS storage for the OS, bandwidth fees and it's quite a bit more.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#285

Post by gshulegaard »

valyala wrote: post #278
gshulegaard wrote: post #153

Earlier quoted context omitted.

Just wanted to say I am super impressed with the work TimescaleDB has been doing. Previously at NGINX I was part of a team that built out a sharded timeseries database using Postgres 9.4. When I left it was ingesting ~2 TB worth of monitoring data a day (so not super large, but not trivial either). Currently I have built out a data warehouse using Postgres 11 and Citus. Only reason I didn't use TimescaleDB was lack o…

Did you try ClickHouse? [1] We were successfully ingesting hundreds of billions of ad serving events per day to it. It is much faster at query speed than any Postgres-based database (for instance, it may scan tens of billions of rows per second on a single node). And it scales to many nodes. While it is possible to store monitoring data to ClickHouse, it may be non-trivial to set up. So we decided creating VictoriaMe…

ClickHouse's intial release was circa 2016 IIRC. The work I was doing at NGINX predates ClickHouse's initial release by 1-2 years.

ClickHouse was certainly something we evaluated later on when we were looking at moving to a true columnar storage approach, but like most columnar systems there are trade-offs.

* Partial SQL support.

* No transactions (not ACID).

* Certain workloads are less efficient like updates and deletes, or single key look ups.

None of these are unique to ClickHouse, they are fairly well known trade-offs most columnar stores make to improve write throughput and prioritize high scaling sequential read performance. As I mentioned before, the amount of data we were ingesting never really reached the limits of even Postgres 9.4, so we didn't feel like we had to make those trade-offs...yet.

I would imagine that servicing ad events is several factors larger scale than we were dealing with.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#286

Post by valyala »

linsomniac wrote: post #111

I've been doing monitoring of our ~120ish machines for 3-4 years now using Influx+Telegraf+Grafana, and have been really happy with it. Prior to that we were using collectd+graphite and with 1 minute stats it was adding some double-digits %age utilization on our infrastructure (I don't remember exactly how much, but I want to say 30% CPU+disk). Influxdb has been a real workhorse. We suffered through some of their ear…

> I really wanted a push rather than pull architecture

Then try VictoriaMetrics - it supports both pull and push (including Influx line protocol) [1], it works out of the box and it requires lower amounts of CPU and RAM when working with big number of time series (aka high cardinality) [2].

[1] https://victoriametrics.github.io/#how-to-import-time-series...

[2] https://medium.com/@valyala/insert-benchmarks-with-inch-infl...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#287

Post by cloudpundit »

freehunter wrote: post #226
bpodgursky wrote: post #121

Earlier quoted context omitted.

Like any SaaS dev tool, when at scale, you negotiate and pay a fraction of the list price. It's meaningless to look at the price of Datadog@5 hosts -- at 500 or 5000, you're paying a completely detached number from the website list price, likely a small fraction.

Which blows away the reason a lot of people/teams/companies like SaaS. Because there is no negotiation, no sales requisitions, no long lead time while they come up with a quote. You see the price you pay the price you get the service, same as anyone else.

Every SaaS provider offers discounts based on length of commitment and volume of spend. Most will say look at AWS, but even there you have list prices and private/bulk pricing. Throw in EDPs, negotiated credits, savings plans, reservations, etc and you're nowhere near list prices.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#288

Post by mavam »

Papric0re wrote: post #74

[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention…

(Disclaimer: CEO & founder of Tenzir)

We at Tenzir are developing VAST for this purpose: https://github.com/tenzir/vast. It's still very early stage, but if you're up for trying something new, a lean and modern C++ architecture, BSD-license open-source style, you may want to give it a spin. The docs are over at https://docs.tenzir.com/vast.

It supports full PCAP, NetFlow, and logs from major security tools. There is CLI and Python bindings. The Apache Arrow bridge offer a high-bandwidth output path into other downstream analytics tools.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#290

Post by valyala »

mekster wrote: post #252
zytek wrote: post #70

VictoriaMetrics eats other TSDBs for breakfast. PromQL support (with extensions) and clustered / HA mode. Great storage efficiency. Plays well for monitoring multiple k8s clusters, works great with Grafana, pretty easily deployed on k8s. No affiliation, just a happy user.

I just don't get why VictoriaMetrics doesn't get more visibility. Maybe they need a PR person.

Absolutely! We are working on this.
Post reply on HN