Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

serhack.me

Post reply on HN
Print view View original

221–230 of 298 posts

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#221

Post by znpy »

Twirrim wrote: post #144
hnarn wrote: post #129

Earlier quoted context omitted.

I haven't, unfortunately, but it looks promising from just looking into it briefly. Open source monitoring is always an area that needs more competition.

Zabbix has been around for quite a long time. Easily 15 years now. I haven't looked at it since around 2013, but at the time it was placing quite some pressure on a mysql db backend. It looks like they've expanded out to support more than MySQL as the back-end these days.

Zabbix has been growing A LOT lately, and in a good way. It's nice to see this king of projects evolving Una good direction instead of stagnating and the diying.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#222

Post by mhall119 »

jakobdabo wrote: post #219

I still can't find any alternative to the old, RRD-based Munin. It is so simple. You want to add a new server to monitor? Just install there the node part, enable any required additional plugins (just by creating a couple of soft-links), add one-line configuration to the main server with the new node's IP address, and you are done. Also, the aesthetics of the UX, you see all the graphs in one single page[1], no addit…

That's more steps than using InfluxDB and Telegraf.

Check out: https://github.com/influxdata/community-templates/tree/maste...

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#223

Post by cevian »

aeyes wrote: post #209
cevian wrote: post #137

Earlier quoted context omitted.

(TimescaleDB engineer here). This is a really unfair critique. The project you cite is super-optimized for the Prometheus use-case and data model. TimescaleDB beats InfluxDB on performance even without these optimization. It's also not possible to optimize in this way in most other time-series databases. These scripts also work hard to give users a UIUX experience that mimicks PromQL in a lot of ways. This isn't nece…

> TimescaleDB beats InfluxDB on performance even without these optimization Would you mind sharing the schema used for this comparison? Maybe I missed it in your documentation of use-cases. When implementing dynamic tags in my own model, my tests showed that your approach is very necessary.

You can look at what we use in our benchmarking tool https://github.com/timescale/tsbs (results described here https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-...).

Pretty much it's a table with time, value, tags_id. Where the tags table is id, jsonb

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#224

Post by sciurus »

bpodgursky wrote: post #121
jakozaur wrote: post #115

Earlier quoted context omitted.

DataDog per host pricing can be very expensive. Metrics are provided by many platforms. If you need logs too, you may look at Sumo Logic which got way cheaper metrics in typical use case. Disclaimer: I work at Sumo Logic.

Like any SaaS dev tool, when at scale, you negotiate and pay a fraction of the list price. It's meaningless to look at the price of Datadog@5 hosts -- at 500 or 5000, you're paying a completely detached number from the website list price, likely a small fraction.

Based on my experience at two companies with thousands of hosts on Datadog, this is not true. You'll only be able to negotiate a small discount.

You'll also experience their habit of launching new features, waiting a while for customers to adopt them, then starting to charge extra for them.

Don't get me wrong, Datadog has great products. But they're also great at extracting money from their customers.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#225

Post by mhall119 »

RedShift1 wrote: post #78
overcast wrote: post #3

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

InfluxDB is pretty good if you don't need to do any advanced querying like grouping by month or formulas. Its strong points are low diskspace footprint and very fast queries even over long periods of time.

Have you looked at Flux? You can do some really incredible things with it, both inside of queries and also in tasks/alerts. Check out https://www.influxdata.com/blog/anomaly-detection-with-media... for an example

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#226

Post by freehunter »

bpodgursky wrote: post #121
jakozaur wrote: post #115

Earlier quoted context omitted.

DataDog per host pricing can be very expensive. Metrics are provided by many platforms. If you need logs too, you may look at Sumo Logic which got way cheaper metrics in typical use case. Disclaimer: I work at Sumo Logic.

Like any SaaS dev tool, when at scale, you negotiate and pay a fraction of the list price. It's meaningless to look at the price of Datadog@5 hosts -- at 500 or 5000, you're paying a completely detached number from the website list price, likely a small fraction.

Which blows away the reason a lot of people/teams/companies like SaaS. Because there is no negotiation, no sales requisitions, no long lead time while they come up with a quote. You see the price you pay the price you get the service, same as anyone else.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#227

Post by gen220 »

doublesCs wrote: post #197
thibauts wrote: post #169

Earlier quoted context omitted.

I’m really sad to see this kind of tone more and more in HN comments...

I find it sadder that people talk such glaring nonsense and are taken seriously. Extremely sad.

IMO, the HN guidelines are pretty clear and reasonable regarding these situations.

> Be kind. Don't be snarky. Have curious conversation; don't cross-examine. Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

The original poster was making an assertion that TDB is too complex. As the guidelines suggest, it doesn't help the conversation to assume that they arrived upon this conclusion unreasonably, and that they're irrational or otherwise unwilling to have an open conversation about it.

We have to assume that they are open to a discussion, despite whatever we might interpret as evidence to the contrary.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#228

Post by BurritoAlPastor »

hnarn wrote: post #163
BurritoAlPastor wrote: post #141

Earlier quoted context omitted.

Nagios is the Jenkins of monitoring. It's popular because you can get it running in an afternoon, and it's easy to configure by hand. It then rots within your infrastructure, because it resists being configured any way _except_ by hand. I've built two systems for configuration-management of Nagios (at different companies), and it's an unpleasant problem to solve. Prometheus's metric format and query syntax are cool,…

Yeah, I was in no way insinuating that Nagios is superior in general, or even to Prometheus, just that it does the job well for some use cases. Monitoring is tricky and you definitely need a tool box because each problem has a different optimal solution.

Nagios and its forks for sure have a place in the monitoring ecosystem. They’re just not tools that tend to stick around once you’re big enough to have a dedicated DevOps or SRE team.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#229

Post by sciurus »

user5994461 wrote: post #143
dijit wrote: post #124

Earlier quoted context omitted.

2,500~ compute instances. Looking at the pricing page the cost $15/instance so $37,500 Middle-of-the-road developer salary is like $35k in most of Europe, outside of the capitals. Although it does say: "Volume discounts available (500+ hosts/mo). Contact us." at the bottom, so I guess 500 is a lot.

2500 instances could be millions per months in AWS costs. The smallest instances with some disks and bandwidth fees can push 100k a month. Spending a fraction of that to monitor that sort of infrastructure is absolutely justified. I can tell you from experience that datadog gives discount even for 100+ hosts, I don't know what they can do for 2500, but if it were me I wouldn't accept anything less than 50% off. Hones…

It could also be less than $100k a month (e.g. 2500 c5a.large with a 1-year reservation). At that point you'd wonder why your monitoring bill was 40% of your compute bill.

Also, of course their salary is relevant. The cost of an engineer's time is an important factor to consider when making build vs buy decisions. Usually it's one that argues in favor of "buy", but not always.

Re: Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

#230

Post by gshulegaard »

roskilli wrote: post #215
gshulegaard wrote: post #153

Earlier quoted context omitted.

Just wanted to say I am super impressed with the work TimescaleDB has been doing. Previously at NGINX I was part of a team that built out a sharded timeseries database using Postgres 9.4. When I left it was ingesting ~2 TB worth of monitoring data a day (so not super large, but not trivial either). Currently I have built out a data warehouse using Postgres 11 and Citus. Only reason I didn't use TimescaleDB was lack o…

If you're looking at scaling monitoring timeseries data you may also wanter to consider more Availability leaning architecture (in the CAP theory sense) with respect to replication (i.e. quorum write/read replication, strictly not leader/follower - active/passive architecture) then you might also want to check out the Apache 2 project M3 and M3DB at m3db.io. I am biased obviously as a contributor. Having said that I…

Neat! Hadn't heard of M3DB before, but cursory poke around the docs seems like it's a pretty solid solution/approach.

My current use case isn't monitoring, or even time series anymore, but will keep M3DB in mind next time I have to seriously push a time series/monitoring solution.

Post reply on HN