Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jaccard Index

en.wikipedia.org

Post reply on HN
Print view View original

21–30 of 46 posts

Re: Jaccard Index

#21

Post by dopidopHN »

B0073D wrote: post #20
pncnmnp wrote: post #14

I recently wrote a fun blog post ( https://pncnmnp.github.io/blogs/odd-sketches.html ) about how to estimate Jaccard Similarity using min hashing, what b-bit min hashing is, and how to improve upon its limitations using a 2014 data structure called odd sketches. Jaccard Similarity's history is also quite interesting. From my blog: > In the late 19th century, the United States and several European nations were focused…

Recently I wanted to try and use this to filter out logs I don't care about, but it seemed a lot more involved than I initially thought. I essentially wanted to use this as a way to flexibly filter out items without having to come up with a regex for every line item. I wonder if anyone has done this before...

I often feel modern tools should offer that

Re: Jaccard Index

#22

Post by reichardt »

ketralnis wrote: post #9

Quoting myself from a while ago[0] At reddit many moons ago before machine learning was a buzzword one early iteration of recommendations was based on Jaccard distance using the number of co-voters between subreddits. But with one twist: divide by the size of the smaller subreddit. relatedness a b = numerator = | voters on(a) ∩ voters on(b) | denominator = | voters on(a) ∪ voters on(b) | weight = min(|voters on(a)|,…

Super interesting! I’m currently writing my master thesis on analyzing relationships between subreddits based on user and semantic similarity. For user similarity I use the Jaccard similarity between the unique set of authors of each subreddit.

Can I send you a message and quote you in my thesis? You can shoot me a short message as well: violets.parr-0c@icloud.com

Re: Jaccard Index

#23

Post by b_mc2 »

I've used this recently to do some fuzzy matching of column names in datasets, I also added it to a small python one-liner library I've been making for practice. p.s. don't give me flack, I know this isn't an efficient way to do things.

jaccard = lambda A, B: len(set(A).intersection(set(B))) / len(set(A).union(set(B)))

https://github.com/b-mc2/MiniMath

Re: Jaccard Index

#24

Post by ta988 »

B0073D wrote: post #20
pncnmnp wrote: post #14

I recently wrote a fun blog post ( https://pncnmnp.github.io/blogs/odd-sketches.html ) about how to estimate Jaccard Similarity using min hashing, what b-bit min hashing is, and how to improve upon its limitations using a 2014 data structure called odd sketches. Jaccard Similarity's history is also quite interesting. From my blog: > In the late 19th century, the United States and several European nations were focused…

Recently I wanted to try and use this to filter out logs I don't care about, but it seemed a lot more involved than I initially thought. I essentially wanted to use this as a way to flexibly filter out items without having to come up with a regex for every line item. I wonder if anyone has done this before...

Bayesian filters like for emails? You mark them as important or noise and over time it will learn. These are extremely easy to put in place and you don't have to preannotate as it learns as you go.

Re: Jaccard Index

#26

Post by ketralnis »

reichardt wrote: post #22
ketralnis wrote: post #9

Quoting myself from a while ago[0] At reddit many moons ago before machine learning was a buzzword one early iteration of recommendations was based on Jaccard distance using the number of co-voters between subreddits. But with one twist: divide by the size of the smaller subreddit. relatedness a b = numerator = | voters on(a) ∩ voters on(b) | denominator = | voters on(a) ∪ voters on(b) | weight = min(|voters on(a)|,…

Super interesting! I’m currently writing my master thesis on analyzing relationships between subreddits based on user and semantic similarity. For user similarity I use the Jaccard similarity between the unique set of authors of each subreddit. Can I send you a message and quote you in my thesis? You can shoot me a short message as well: violets.parr-0c@icloud.com

Yeah sure, you can message me on reddit too. Same username

Re: Jaccard Index

#27

Post by zamadatix »

B0073D wrote: post #20
pncnmnp wrote: post #14

I recently wrote a fun blog post ( https://pncnmnp.github.io/blogs/odd-sketches.html ) about how to estimate Jaccard Similarity using min hashing, what b-bit min hashing is, and how to improve upon its limitations using a 2014 data structure called odd sketches. Jaccard Similarity's history is also quite interesting. From my blog: > In the late 19th century, the United States and several European nations were focused…

Recently I wanted to try and use this to filter out logs I don't care about, but it seemed a lot more involved than I initially thought. I essentially wanted to use this as a way to flexibly filter out items without having to come up with a regex for every line item. I wonder if anyone has done this before...

Could always rely on the Levenshtein distance. You have to be careful with similarity approaches though as you may end up filtering important messages because they are structurally similar to the unimportant message.

Re: Jaccard Index

#29

Post by bitshiftfaced »

One of the weaknesses with Jaccard similarity is how it focuses on matches/true positives. It neglects the importance of "negative space."

I was happy to see Matthew's correlation coefficient (MCC) used in the recent "1st and Future - Player Contact Detection" Kaggle competition. MCC balances the eight confusion matrix ratios, and I've gotten excellent results when using it in the past.

Re: Jaccard Index

#30

Post by aquafox »

psyklic wrote: post #2

This is one of my favorite distance metrics* to show people! For example, perhaps one person likes Reddit and HN, while someone else likes HN and SO. Then their Jaccard Index would be 1/3, since they have one thing in common out of three. * Technically it computes "similarity" (larger number == more similar), but `1 - Jaccard Index` is a distance (smaller number == more similar).

Very useful indeed. I used it to compute the similarity of patients with different diseases, e.g. if patient1 has 3 diseases and patient2 has 2 of those three, they should be treated more similar than if patient1 has only 1 disease and patient2 has another one. Euclidean distance would assign the same value in both cases.
Post reply on HN