21–30 of 61 posts
have you considered a deterministic tier before the embedding pass? I feel that approach can be more efficient.
If by "more efficient" you mean to avoid embedding of the same code multiple times, this optimization is already implemented internally.
Does the project you simply compute embeddings for every function unit and cluster them, or do we also mean-pool significant dependencies of a function? In other words, given the function
def a():
b()
c()
d()
have you considered a deterministic tier before the embedding pass? I feel that approach can be more efficient.
This is a great use case for embeddings. Code deduplication across distant modules is notoriously hard for traditional AST-based tools. How do you handle chunking and parsing for different languages to make sure the embeddings capture semantic meaning effectively? For instance, do you chunk by functions/classes, or use a fixed token window? If a function is too long or too short, it can drastically skew the embedding…
The chunk size has allowed range and those outside are simply ignored.
- Upper limit is hardcoded with a body size of 10k chars
- Lower limit is configurable with a default of 10 AST nodes inside the body
The chunking strategy is something that can be improved in future versions.
Cool project, I've been meaning to do this myself at work for a codebase, and it's nice to see that this exists now. Does the project you simply compute embeddings for every function unit and cluster them, or do we also mean-pool significant dependencies of a function? In other words, given the function def a(): b() c() d() Do we also embed b, c, and d as well and combine them somehow in the embedding of a?
[1] https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexi...
I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans. It finds similar-looking code with embeddings. This detects more than just copy-paste clones or even clones with minor changes. Similar code is often not a clone to refactor, and this is a trade-off. Initial results need to be verified, but coding agents can do this quickly. Exampl…
Cool project, I've been meaning to do this myself at work for a codebase, and it's nice to see that this exists now. Does the project you simply compute embeddings for every function unit and cluster them, or do we also mean-pool significant dependencies of a function? In other words, given the function def a(): b() c() d() Do we also embed b, c, and d as well and combine them somehow in the embedding of a?
I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans. It finds similar-looking code with embeddings. This detects more than just copy-paste clones or even clones with minor changes. Similar code is often not a clone to refactor, and this is a trade-off. Initial results need to be verified, but coding agents can do this quickly. Exampl…