31–40 of 46 posts
The Jaccard similarity between sets of uni- and bi-grams was a surprising effective metric.
DOG -> {d, o, g, do, og}
GOD -> {g, o, d, go, od}
intersection = {d, g, o}
union = {d, g, o, do, go, od, og}
J = 3 / 7 = ~43%
I recently wrote a fun blog post ( https://pncnmnp.github.io/blogs/odd-sketches.html ) about how to estimate Jaccard Similarity using min hashing, what b-bit min hashing is, and how to improve upon its limitations using a 2014 data structure called odd sketches. Jaccard Similarity's history is also quite interesting. From my blog: > In the late 19th century, the United States and several European nations were focused…
Recently I wanted to try and use this to filter out logs I don't care about, but it seemed a lot more involved than I initially thought. I essentially wanted to use this as a way to flexibly filter out items without having to come up with a regex for every line item. I wonder if anyone has done this before...
Earlier quoted context omitted.
Recently I wanted to try and use this to filter out logs I don't care about, but it seemed a lot more involved than I initially thought. I essentially wanted to use this as a way to flexibly filter out items without having to come up with a regex for every line item. I wonder if anyone has done this before...
Bayesian filters like for emails? You mark them as important or noise and over time it will learn. These are extremely easy to put in place and you don't have to preannotate as it learns as you go.
I had an idea Splunk had them built in? But it's about 5 lines of Python anyway.
[1] http://www.mmds.org/ Chapter 3 [2] https://arxiv.org/abs/1706.05698
Earlier quoted context omitted.
Recently I wanted to try and use this to filter out logs I don't care about, but it seemed a lot more involved than I initially thought. I essentially wanted to use this as a way to flexibly filter out items without having to come up with a regex for every line item. I wonder if anyone has done this before...
Could always rely on the Levenshtein distance. You have to be careful with similarity approaches though as you may end up filtering important messages because they are structurally similar to the unimportant message.
Of note: the author of that library is none other than Daniel Lemire [3], whose articles pop up quite often on HN
[1] https://roaringbitmap.org/
[2] https://roaringbitmap.readthedocs.io/en/latest/#roaringbitma...
Reflection coefficient in electrical or acoustic (or elastic) transmission across two media is the difference of their impedances over the sum of them.
Difference over sum is a pattern you see a lot.
One of the weaknesses with Jaccard similarity is how it focuses on matches/true positives. It neglects the importance of "negative space." I was happy to see Matthew's correlation coefficient (MCC) used in the recent "1st and Future - Player Contact Detection" Kaggle competition. MCC balances the eight confusion matrix ratios, and I've gotten excellent results when using it in the past.
One that makes it the better choice in situations where negative space should in fact be ignored. (comparing chest xrays are a typical example in medical imaging)