Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for detecting non-exact code duplication with embedding models

github.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#51

Post by SpyCoder77 »

rufius wrote: post #14
SpyCoder77 wrote: post #10

I think that this is pretty cool, but is there any reason why we would want to remove similar/possible duplicate code?

(without sarcasm) Is this a serious question? If so - maintainability, testability. This is old software engineering best practice at this point. You shouldn’t hyper optimize for deduplication, but it’s usually worth considering. Fewer places to fix issues or improve as well.

I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#52

Post by SpyCoder77 »

Zopieux wrote: post #13
SpyCoder77 wrote: post #10

I think that this is pretty cool, but is there any reason why we would want to remove similar/possible duplicate code?

Have you written software before?

I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#53

Post by msephton »

rkochanowski wrote: post #42
msephton wrote: post #39

How does it compare to jscpd? https://github.com/kucherenko/jscpd

It's the opposite. jscpd is advertised as "Copy/paste detector", Slopo is advertised as "non-exact code duplication". Slopo also detects copy/pasted code, but this is not the main goal and the report focuses more on similar code units.

I've used jscpd, it does "non-exact code duplication" (regardless of what its readme says) which is why I asked how Slopo compares. I'm surprised you've not tried the competition!

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#55

Post by rkochanowski »

ahmetson wrote: post #54

For false positives, how about generating the test units and run it in isolation? All major languages have the interpreters or embedded languages to execute function only.

Do you mean to verify if similar code units produce the same result?

The goal of the tool is to also detect code that is similar and behaves differently. There are not ideal duplicates, but still a code that can be refactored, abstracted, or fixed (because the variance may be the result of bug).

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#56

Post by Bibabomas »

rkochanowski wrote: post #49
Bibabomas wrote: post #45

Earlier quoted context omitted.

I don't understand what you mean with your first sentence. Both SemHash and Slopo are deduplication libraries right? Regardless, finding similar code is the core functionality that enables (semantic) deduplication. I also don't think the backend is the "easiest part" here, a lot of the scalability lives there, which is important for monorepos, or cases where you want to deduplicate across projects. For example, from…

Slopo doesn't deduplicate. It only reports similar code units, which in most cases are not duplicates. This needs to be cleaned up by the user, which is fast and easy with coding agents. Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper. "brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects,…

"it works fine and this is not a bottleneck. For large projects, other parts are much slower": I don't think this is true, especially for large projects. I just ran your tool on the Kubernetes repo with 1536-dim embeddings. The isolated similar-pair search took ~130s and peaked at ~4.8 GB RSS, and the total runtime was ~250s with the same memory peak.

"In larger sets, all vectors are not loaded into memory at once": this is also not correct, at least in the implementation you shared. The similarity matrix is processed in blocks, but the embeddings themselves are loaded all at once and stacked into one NumPy matrix, hence the memory peak.

So in larger projects, more than half of the time is spent on the similarity search step, and almost all the memory is spent there as well.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#57

Post by rkochanowski »

msephton wrote: post #53
rkochanowski wrote: post #42

Earlier quoted context omitted.

It's the opposite. jscpd is advertised as "Copy/paste detector", Slopo is advertised as "non-exact code duplication". Slopo also detects copy/pasted code, but this is not the main goal and the report focuses more on similar code units.

I've used jscpd, it does "non-exact code duplication" (regardless of what its readme says) which is why I asked how Slopo compares. I'm surprised you've not tried the competition!

"Non-exact" may be interpreted in different ways, so let's look at the example report: https://github.com/rafal-qa/slopo/tree/main/doc/example-repo...

* cluster-01.md has the highest similarity, and jscpd probably will detect this too.

* cluster-10.md has the lowest similarity still above the threshold, and I don't think jscpd will detect this as clones. Because they are not clones, this is a false positive that needs to be discarded. But in other cases this kind of similar code may be worth acting on.

I didn't compare with jscpd because I don't consider it a competition. Embedding-based duplication detection works differently, gives different results and has its own trade-offs.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#58

Post by rkochanowski »

Bibabomas wrote: post #56
rkochanowski wrote: post #49

Earlier quoted context omitted.

Slopo doesn't deduplicate. It only reports similar code units, which in most cases are not duplicates. This needs to be cleaned up by the user, which is fast and easy with coding agents. Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper. "brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects,…

"it works fine and this is not a bottleneck. For large projects, other parts are much slower": I don't think this is true, especially for large projects. I just ran your tool on the Kubernetes repo with 1536-dim embeddings. The isolated similar-pair search took ~130s and peaked at ~4.8 GB RSS, and the total runtime was ~250s with the same memory peak. "In larger sets, all vectors are not loaded into memory at once":…

Thanks, I will look at this in more detail.

To give more context, the current version is already an optimized one I considered good enough and didn't spend more time on it. In the first attempt, I used a vector database with indexes, trying to query for similar vectors. This was uselessly slow even in medium-sized repos. The brute-force NumPy solution is a significant improvement, making it faster than other calculations like clustering.

"vectors are not loaded into memory at once" is not true, I had in mind splitting computation into blocks.

One possible simple optimization is to use 16-bit floats in vector instead of 32-bit. I used this in a different project (halfvec in pgvector) without affecting results.

1536-dim embeddings from your case also can be reduced. This large vector usually doesn't give much benefit compared to smaller ones. And this is something I will compare in my own tests.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#60

Post by msephton »

rkochanowski wrote: post #57
msephton wrote: post #53

Earlier quoted context omitted.

I've used jscpd, it does "non-exact code duplication" (regardless of what its readme says) which is why I asked how Slopo compares. I'm surprised you've not tried the competition!

"Non-exact" may be interpreted in different ways, so let's look at the example report: https://github.com/rafal-qa/slopo/tree/main/doc/example-repo... * cluster-01.md has the highest similarity, and jscpd probably will detect this too. * cluster-10.md has the lowest similarity still above the threshold, and I don't think jscpd will detect this as clones. Because they are not clones, this is a false positive that need…

[flagged]
Post reply on HN