Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for detecting non-exact code duplication with embedding models

github.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#41

Post by rkochanowski »

Bibabomas wrote: post #40

Have you compared this to https://github.com/MinishLab/semhash (or considered using that for the deduplication backend)?

Finding similar code is something different than deduplication, even when the final goal looks similar.

Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around.

Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, including local. I left this part for the user.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#42

Post by rkochanowski »

msephton wrote: post #39

How does it compare to jscpd? https://github.com/kucherenko/jscpd

It's the opposite.

jscpd is advertised as "Copy/paste detector", Slopo is advertised as "non-exact code duplication".

Slopo also detects copy/pasted code, but this is not the main goal and the report focuses more on similar code units.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#44

Post by rkochanowski »

romanoonhn wrote: post #37

Looks very cool! I'd be very interested in applying this to my Elixir projects. What does it take to add proper support for a new language?

It can be added with https://pypi.org/project/tree-sitter-elixir/ similar to other languages. I will add this and plan to release a new version today.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#45

Post by Bibabomas »

rkochanowski wrote: post #41
Bibabomas wrote: post #40

Have you compared this to https://github.com/MinishLab/semhash (or considered using that for the deduplication backend)?

Finding similar code is something different than deduplication, even when the final goal looks similar. Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around. Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, i…

I don't understand what you mean with your first sentence. Both SemHash and Slopo are deduplication libraries right? Regardless, finding similar code is the core functionality that enables (semantic) deduplication.

I also don't think the backend is the "easiest part" here, a lot of the scalability lives there, which is important for monorepos, or cases where you want to deduplicate across projects. For example, from looking at the implementation, you use exact brute-force similarity search (comparing every item to every other item) which is an O(n^2) operation. It also allocates large dense similarity blocks in memory, so memory use won’t scale well either.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#46

Post by AlexeyBelov »

rkochanowski wrote: post #34
nttylock wrote: post #27

Earlier quoted context omitted.

[flagged]

My solution for false positives is simpler: 1. The tool uses only cosine similarity plus boost depending on distance in the codebase. 2. Classification with LLM. This can be done by coding agent used with project giving better results than integrating this pass in the tool. LLMs used for coding are pretty good. I assumed that this is not a problem I need to solve inside the tool. I'm aware this is not deterministic,…

You're replying to an LLM bot.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#48

Post by rkochanowski »

realxrobau wrote: post #3
rkochanowski wrote: post #2

I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans. It finds similar-looking code with embeddings. This detects more than just copy-paste clones or even clones with minor changes. Similar code is often not a clone to refactor, and this is a trade-off. Initial results need to be verified, but coding agents can do this quickly. Exampl…

If it did PHP I would love to run it over WordPress. What would it take to add that?

PHP was added in the latest release.

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#49

Post by rkochanowski »

Bibabomas wrote: post #45
rkochanowski wrote: post #41

Earlier quoted context omitted.

Finding similar code is something different than deduplication, even when the final goal looks similar. Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around. Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, i…

I don't understand what you mean with your first sentence. Both SemHash and Slopo are deduplication libraries right? Regardless, finding similar code is the core functionality that enables (semantic) deduplication. I also don't think the backend is the "easiest part" here, a lot of the scalability lives there, which is important for monorepos, or cases where you want to deduplicate across projects. For example, from…

Slopo doesn't deduplicate. It only reports similar code units, which in most cases are not duplicates. This needs to be cleaned up by the user, which is fast and easy with coding agents.

Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper.

"brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects, other parts are much slower, which has room for improvement. [1] is an implementation you probably saw. It uses NumPy, spreads work across all CPU cores and there is also a split into blocks (block_size = 1000). In larger sets, all vectors are not loaded into memory at once. Where I need to be honest, I didn't measure actual memory usage. I just tested this on large repos, so I'm aware of bottlenecks.

From my perspective, this is the easiest part. Code extraction, chunking, applying boost, clustering, generating report and designing everything as a single user-friendly tool is a real challenge. Architectural and product decisions are more difficult than implementation and solving performance issues.

[1] https://github.com/rafal-qa/slopo/blob/v0.3.0/src/slopo/anal...

Re: Show HN: CLI tool for detecting non-exact code duplication with embedding models

#50

Post by SpyCoder77 »

rkochanowski wrote: post #17
SpyCoder77 wrote: post #10

I think that this is pretty cool, but is there any reason why we would want to remove similar/possible duplicate code?

Recently there was a popular article on HN saying that sometimes code duplication is better than abstraction, so I assume that this question is not a joke. While testing this tool, one detected duplication was interesting for a use case. Permission check logic was duplicated and placed in different distant places in the codebase. The code was similar, but not identical, the logic was not the same. One version had str…

I seem to have misunderstood. I thought that it was talking about stuff like similar repos, but now I realize that this is most likely talking about a singular codebase
Post reply on HN