Why would I use it over DataDog?
Different scale? Different price point?
Netdata: Open-source real-time monitoring platform
41–50 of 108 posts
Re: Netdata: Open-source real-time monitoring platform
#42It’s very neat for individual servers Doesn’t work well for monitoring multiple servers though from what I can tell.
There's a netdata prometheus exporter, but it overlaps a lot with node-exporter. If you're already runnning netdata however then it could be a good choice
node_exporter is a lot more robust. Had both running for a while and netdata would get stuck when there's I/O trouble, while node_exporter was carefully built not to do any I/O and kept working just fine.
Re: Netdata: Open-source real-time monitoring platform
#43Note that netdata phones home without consent in the default configuration. For many, the whole point of doing system-administration is selfhosting and autonomy, and privacy is frequently a big component of that.
Netdata blows a big hole in that by transmitting your usage information off of your box without getting permission.
Re: Netdata: Open-source real-time monitoring platform
#44Earlier quoted context omitted.
Disclaimer - i work at Netdata Cloud on ML. This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user that the user can then quickly digest and decide if they are or are not something that could represent an "incident". So human in the loop sort of ML to help assist and lower the cognitive load of all the charts. We have a first step on this ladder via the pytho…
What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in practice (especially at large scale, where you'll get spurious correlations). What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
It's about troubleshooting. When you have a complex infrastructure, it's not enough to say that your db queries are slower than usual. Ok, so you immediately see that your db server is getting a lot more traffic. What was the root cause though and what can you do about it now? Given enough "funny charts", you can see for example that you have hit a resource limit that you can temporarily raise and also see that a particular component of your infrastructure has an anomalous behavior, e.g. a cron job that was usually utilizing resources for a few seconds, now takes minutes. So you can provide a quick workaround and move on to investigate what changed with that cron job.
Re: Netdata: Open-source real-time monitoring platform
#45How does it compare to New Relic who also happens to monitor, if enabled, containers and system things?
Re: Netdata: Open-source real-time monitoring platform
#46Could someone enlighten me on the internals, how is netdata able to get realtime granularity, whereas prometheus defaults to 15s?
It's because it was built with high granularity and unlimited metrics as a key differentiator from the beginning. The core is written in pure C, optimized to death. Even long-term retention was initially sacrificed, in order to be able achieve that high performance, with minimal resource needs.
Long term retention is now possible, but with relatively high memory requirements, depending on how many metrics are collected. Again, it was a decision to never give up realtime granularity and speed, even at the cost of writing our own timeseries db in C and utilizing more memory.
Re: Netdata: Open-source real-time monitoring platform
#47Earlier quoted context omitted.
Disclaimer - i work at Netdata Cloud on ML. This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user that the user can then quickly digest and decide if they are or are not something that could represent an "incident". So human in the loop sort of ML to help assist and lower the cognitive load of all the charts. We have a first step on this ladder via the pytho…
What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in practice (especially at large scale, where you'll get spurious correlations). What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
Do 4xx responses count against your SLO? For me they don't, but an abnormal increase might still signify that something is actually wrong. (I haven't yet found a useful tool for highlighting this kind of abnormality though)
Re: Netdata: Open-source real-time monitoring platform
#48Earlier quoted context omitted.
So the only convenient way to have security is to use the cloud version? Got it.
yes, because a 10 line nginx config with basic http auth is too difficult for a sysadmin to set up in conjunction with his systems monitoring tool stop being obtuse
It's not that it's too difficult, but we were accustomed to having this functionality built in in similar products in the past, then things changed. When ELK first showed up there was a big wave of attacks on ELK servers because they were completely unsecured and at that time X-Pack Security was a paid add-on, they changed their mind later, some time after an open source solution appeared.
Re: Netdata: Open-source real-time monitoring platform
#49Note that netdata phones home without consent in the default configuration. For many, the whole point of doing system-administration is selfhosting and autonomy, and privacy is frequently a big component of that. Netdata blows a big hole in that by transmitting your usage information off of your box without getting permission.
That's a bit unfair. In the docs we are being very upfront that you can opt-out of anonymous telemetry: https://learn.netdata.cloud/docs/get
we use the data we gather in order to make smarter product decisions. We want to invest resources where it matters, so we need to know how our users use the product.
We are also very detailed on what we gather: https://learn.netdata.cloud/docs/agent/netdata-security
Lastly, we just changed our analytics engine, from google-analytics to a self-hosted posthog, which is an open-source product analytics platform
Re: Netdata: Open-source real-time monitoring platform
#50Earlier quoted context omitted.
Disclaimer - i work at Netdata Cloud on ML. This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user that the user can then quickly digest and decide if they are or are not something that could represent an "incident". So human in the loop sort of ML to help assist and lower the cognitive load of all the charts. We have a first step on this ladder via the pytho…
What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in practice (especially at large scale, where you'll get spurious correlations). What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
I generally agree that well defined SLOs for back end services works as you define service contacts between services and care less about the particular funny graph being surfaced more that a particular service is out of contract.
Where automatic anomaly detection was very valuable for us was in the video domain with multi dimensional end user telemetry. I.e what would be lost as noise in top level metics could be surfaced via anomaly detection for specific combination of dimensions that you could not otherwise manually observe. I.e video start time in Mexico is fine ... But an ISP in Mexico City is not failing but when data is sliced and anomaly highlighted we see its newly under preforming and we need to feed this data into our CDN switching to improve video start time there.
The data had too many dimensions that were always changing with degraded experience easily lost in the noise when measuring across platform and our software updates, combinations target devices, connection types, geo location, specific content, active ab tests, etc. In such cases automatic anomaly detection was pretty critical.