Mute uninteresting log noise with machine learning
blog.machinebox.io
Mute uninteresting log noise with machine learning
1–10 of 19 posts
Re: Mute uninteresting log noise with machine learning
#2Awesome
Re: Mute uninteresting log noise with machine learning
#3Today's uninteresting log noise is tomorrow's critical data.
I've been loving Kibana for filtering and reporting on log data in flexible and insightful ways, including automatically generated charts for certain data sources.
Re: Mute uninteresting log noise with machine learning
#4Is it possible to make a ML algorithm which has only "noise" data for training and then identifies abnormalities? It seems like that's people do that easily and it would be ideal for an application like this where you might not have much training data on all the "not noise" types of examples.
Another application would be a security camera that detects unusual events without having to train it on actual burglars.
Re: Mute uninteresting log noise with machine learning
#5One of the ways that I do this (assuming you have access to unix utilities) is to do:
cat output.log | tr -d '[0-9]' | sort | uniq -c | sort -n
Re: Mute uninteresting log noise with machine learning
#6Is it possible to make a ML algorithm which has only "noise" data for training and then identifies abnormalities? It seems like that's people do that easily and it would be ideal for an application like this where you might not have much training data on all the "not noise" types of examples. Another application would be a security camera that detects unusual events without having to train it on actual burglars.
This is a subfield called anomaly detection.
Re: Mute uninteresting log noise with machine learning
#7One of the ways that I do this (assuming you have access to unix utilities) is to do: cat output.log | tr -d '[0-9]' | sort | uniq -c | sort -n This is a fairly useful way of removing relatively useless information such as timestamps and line numbers when you're looking for rare or unique events. The alternative, I think, is to do a bunch of awk or sed magic, which isn't really fun for anybody. It's especially useful…
onestly I found really fun to do "a bunch of awk or sed magic"
Re: Mute uninteresting log noise with machine learning
#8I have never read useful log output that wasn't generated by code that I had written.
Other people's logs are mostly noxious CPU exhaust.
Re: Mute uninteresting log noise with machine learning
#9It would be neat if MachineBox could sense whether log noise would be useful in other contexts--e.g., as a metric that can be graphed. Or whether your logging is lacking something that might be useful, or just lacking signal at all (hey, user, your logs are just noise!).
Re: Mute uninteresting log noise with machine learning
#10Today's uninteresting log noise is tomorrow's critical data. I've been loving Kibana for filtering and reporting on log data in flexible and insightful ways, including automatically generated charts for certain data sources.
Yes. Using loglevels/priorities, facilities and identities in a sane way, your logs would already be classified.
Let's say there's a service failure and I want to know what the service has done prior to the failure. I wouldn't want a classifier to filter the logs in that case, so that use case is out of the picture. What other use cases than filtering are there for this? Maybe as a way to provide feedback to developers to fix the log messages, as in "this thing that we log all the time can be determined to never affect the process of trouble-shooting our services, and the classifier thinks it's noise, so we'll remove it".