Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jaccard Index

en.wikipedia.org

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Jaccard Index

#41

Post by bitshiftfaced »

tpoacher wrote: post #39
bitshiftfaced wrote: post #29

One of the weaknesses with Jaccard similarity is how it focuses on matches/true positives. It neglects the importance of "negative space." I was happy to see Matthew's correlation coefficient (MCC) used in the recent "1st and Future - Player Contact Detection" Kaggle competition. MCC balances the eight confusion matrix ratios, and I've gotten excellent results when using it in the past.

It's not a weakness; it's a feature. One that makes it the better choice in situations where negative space should in fact be ignored. (comparing chest xrays are a typical example in medical imaging)

It's unclear to me why they should be ignored.

Re: Jaccard Index

#42

Post by john-titor »

this is well-known to people working in cheminfornatics as the tanimoto similarity, used to calculate the similarity of two chemical structures based on their (folded) fingerprint

Re: Jaccard Index

#43

Post by tpoacher »

bitshiftfaced wrote: post #41
tpoacher wrote: post #39

Earlier quoted context omitted.

It's not a weakness; it's a feature. One that makes it the better choice in situations where negative space should in fact be ignored. (comparing chest xrays are a typical example in medical imaging)

It's unclear to me why they should be ignored.

Consider two xrays and the output of an algorithm that tries to outline the lungs.

Suppose that one xray was taken in a larger machine, and therefore has more negative space around the lungs.

Also suppose that the algorithm delineated the lungs equally well in both cases (anatomically speaking).

If you assess performance using the jaccard index, the metric is equal in both cases, as it should be, indicating equal performance of the algorithm w.r.t. the ground truth.

Whereas anything that takes accuracy of true negatives into account will necessarily give a higher performance in the xray from the larger machine, even if the person xrayed and the lung outline were identical.

Re: Jaccard Index

#44

Post by bitshiftfaced »

tpoacher wrote: post #43
bitshiftfaced wrote: post #41

Earlier quoted context omitted.

It's unclear to me why they should be ignored.

Consider two xrays and the output of an algorithm that tries to outline the lungs. Suppose that one xray was taken in a larger machine, and therefore has more negative space around the lungs. Also suppose that the algorithm delineated the lungs equally well in both cases (anatomically speaking). If you assess performance using the jaccard index, the metric is equal in both cases, as it should be, indicating equal per…

I don't see how this is an improvement over MCC in this example, since in this case, unless I'm mistaken, MCC would hypothetically give the same (perfect) value to both x-rays, just as Jaccard would.

Re: Jaccard Index

#45

Post by tpoacher »

bitshiftfaced wrote: post #44
tpoacher wrote: post #43

Earlier quoted context omitted.

Consider two xrays and the output of an algorithm that tries to outline the lungs. Suppose that one xray was taken in a larger machine, and therefore has more negative space around the lungs. Also suppose that the algorithm delineated the lungs equally well in both cases (anatomically speaking). If you assess performance using the jaccard index, the metric is equal in both cases, as it should be, indicating equal per…

I don't see how this is an improvement over MCC in this example, since in this case, unless I'm mistaken, MCC would hypothetically give the same (perfect) value to both x-rays, just as Jaccard would.

No, in this scenario, the MCC will generally give a misleadingly higher value to the larger machine, since it takes into account the higher number of true negatives. In this scenario, this makes it a bad metric.

Obviously, a 100% perfect segmentation would of course register as perfect in both, but in practice one rarely deals with such perfect predictions.

In general, there is no metric that is universally "better" in all scenarios. One is expected to choose the metric that best suits the particular goal one wishes to validate against.

Re: Jaccard Index

#46

Post by d23 »

ketralnis wrote: post #9

Quoting myself from a while ago[0] At reddit many moons ago before machine learning was a buzzword one early iteration of recommendations was based on Jaccard distance using the number of co-voters between subreddits. But with one twist: divide by the size of the smaller subreddit. relatedness a b = numerator = | voters on(a) ∩ voters on(b) | denominator = | voters on(a) ∪ voters on(b) | weight = min(|voters on(a)|,…

Ha, I came here to mention it and of course you’re already here explaining the background in far more detail than I could :)
Post reply on HN