Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sem: New primitive for code understanding – not LSPs, but entities on top of Git

ataraxy-labs.github.io

Post reply on HN
Print view View original

41–50 of 67 posts

Re: Sem: New primitive for code understanding – not LSPs, but entities on top of Git

#41

Post by cpard »

gwerbin wrote: post #37
rohanucla wrote: post #35

Earlier quoted context omitted.

Thanks! The data artifacts angle is really interesting. in some ways the problem is even harder there because data pipelines have less explicit structure than code, I guess.

The artifacts themselves have more structure, but diffing is hard because of size: what exactly do you show in the different? Row-level? Summary statistics? How do you keep it from getting slow on bigger datasets? Then there are plots saved as images which have basically no structure at all exposed.

Row level and summary stats are both diffs over values that can tell you that something changed but not whether the * meaning * has changed. What I'm working on is providing more information on how the meaning changes.

What questions I'd like to answer with the diffing is more like: will the grain go from one-row-per-user to one-row-per-user-per-day, will a key stop being unique, will a join start fanning out and quietly double a measure, will something additive become non-additive.

This diff is over structure but this structure is latent in the transformation that produces it and to make things harder, if we are talking about some declarative language being used (e.g. SQL) the code doesn't even describe how things are getting done, but what the output would be.

What I've ended up doing is recovering the structure from the code by analyzing it and then using * cheap * profiling than a full row compare.

As an example, my equivalent impact sub-command output would be something like this: "this change makes account_id non-unique three models downstream"

Re: Sem: New primitive for code understanding – not LSPs, but entities on top of Git

#42

Post by globnomulous »

Interesting idea. How does it, or how should it, perform in huge monorepos, where git performance suffers? I spend most of my time in a repo that contains hundreds of thousands of files, where just a simple `git status` can take >3.5 seconds even on very fast consumer hardware. (Thank God for sparse-checkout.)

Re: Sem: New primitive for code understanding – not LSPs, but entities on top of Git

#43

Post by rohanucla »

globnomulous wrote: post #42

Interesting idea. How does it, or how should it, perform in huge monorepos, where git performance suffers? I spend most of my time in a repo that contains hundreds of thousands of files, where just a simple `git status` can take >3.5 seconds even on very fast consumer hardware. (Thank God for sparse-checkout.)

This is actually the exact scenario we just spent the last few weeks optimizing for. On a 71K-file TypeScript monorepo, sem was previously choking entirely (DNF), and now completes in 6.5s with the topology cache warm. On a 100K-file generated fixture, sem impact went from 90s cold down to about 1s warm. The key was building a SQLite-backed cache that stores the dependency graph structure so repeat runs skip re-parsing unchanged files entirely.

Re: Sem: New primitive for code understanding – not LSPs, but entities on top of Git

#48

Post by rohanucla »

znnajdla wrote: post #47

Looks very useful, but incredibly obnoxious that it overrides default git diff. And the only way to get the regular git diff output is to uninstall it? There is no way I would ever want to do that.

sem doesn't override git diff, it's a completely separate command (sem diff). Your regular git diff should work exactly as it always has after installing sem.

If you want to change your git diff default behavior then you can do sem setup.

Re: Sem: New primitive for code understanding – not LSPs, but entities on top of Git

#49

Post by rohanucla »

mcintyre1994 wrote: post #45

This looks really neat, but I see its diff output as complimentary to git and wouldn’t want to replace git diff. Is there a way to just install the CLI and MCP server but not override git diff?

It doesn't override git diff at all, sem is its own standalone CLI. git diff continues to work exactly as before. You do sem setup only when you want to change your default git diff behavior, other wise after installing sem you can use it straight away using sem commands.

Re: Sem: New primitive for code understanding – not LSPs, but entities on top of Git

#50

Post by znnajdla »

rohanucla wrote: post #48
znnajdla wrote: post #47

Looks very useful, but incredibly obnoxious that it overrides default git diff. And the only way to get the regular git diff output is to uninstall it? There is no way I would ever want to do that.

sem doesn't override git diff, it's a completely separate command (sem diff). Your regular git diff should work exactly as it always has after installing sem. If you want to change your git diff default behavior then you can do sem setup.

That’s not clear at all from the docs. It shouldn’t be called “setup” then. Even after doing sem setup there should be a CLI flag to get the default diff output without unsetting up. Very annoying hijack.
Post reply on HN