[deleted]
Mute uninteresting log noise with machine learning
11–19 of 19 posts
Re: Mute uninteresting log noise with machine learning
#12I really wanted this to be about real time sound editing and not about log data.
Re: Mute uninteresting log noise with machine learning
#13Would not false negative (a critical log being muted) be a major concern while using machine learning in this domain?
What if I never see a critical log because the trained model decided that it is unimportant? How is such a situation generally solved in the industry?
Re: Mute uninteresting log noise with machine learning
#14Maybe an easier way to go is to record it structured up front (it’s already structured in the original application source anyway). This makes it much easier to record efficiently (so you can record more data) and also much easier to query efficiently, where eg you might invest time in machine learning on logical data instead of having to mess around with text.
That’s what we do here anyway, it’s worked well for us:
https://github.com/Morgan-Stanley/hobbes/blob/master/README....
Re: Mute uninteresting log noise with machine learning
#15Would not false negative (a critical log being muted) be a major concern while using machine learning in this domain? What if I never see a critical log because the trained model decided that it is unimportant? How is such a situation generally solved in the industry?
I have limited experience, but I think that usually you would take this into account when building your loss function and heavily penalize false negatives during training.
Re: Mute uninteresting log noise with machine learning
#16One of the ways that I do this (assuming you have access to unix utilities) is to do: cat output.log | tr -d '[0-9]' | sort | uniq -c | sort -n This is a fairly useful way of removing relatively useless information such as timestamps and line numbers when you're looking for rare or unique events. The alternative, I think, is to do a bunch of awk or sed magic, which isn't really fun for anybody. It's especially useful…
It would be nice if you told us what this command does for you.
Re: Mute uninteresting log noise with machine learning
#17One of the ways that I do this (assuming you have access to unix utilities) is to do: cat output.log | tr -d '[0-9]' | sort | uniq -c | sort -n This is a fairly useful way of removing relatively useless information such as timestamps and line numbers when you're looking for rare or unique events. The alternative, I think, is to do a bunch of awk or sed magic, which isn't really fun for anybody. It's especially useful…
It would be nice if you told us what this command does for you.
It removes numbers from a log file, sorts it, groups and counts unique lines, then sorts numerically by the count of each unique line.
But don't take my word for it. Try it yourself!
Re: Mute uninteresting log noise with machine learning
#18One of the ways that I do this (assuming you have access to unix utilities) is to do: cat output.log | tr -d '[0-9]' | sort | uniq -c | sort -n This is a fairly useful way of removing relatively useless information such as timestamps and line numbers when you're looking for rare or unique events. The alternative, I think, is to do a bunch of awk or sed magic, which isn't really fun for anybody. It's especially useful…
onestly I found really fun to do "a bunch of awk or sed magic"
Except for us weirdos :p
Re: Mute uninteresting log noise with machine learning
#19It would be nice if logfile analysis tools (including ELK) supported logs that were multiple lines per message. Does anyone know of such tools?