Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

llamaindex.ai

Post reply on HN
Print view View original

1–10 of 13 posts

Re: Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

#2

Post by lyjackal »

I recently did something similar. Using uv workspaces, I used the uv CLI's dependency graph to analyze the dependency tree then conditionally trigger CI workflows for affected projects. I wish there was a better way to access the uv dependency worktree other than parsing the `tree` like output

Re: Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

#3

Post by cheesyFish »

lyjackal wrote: post #2

I recently did something similar. Using uv workspaces, I used the uv CLI's dependency graph to analyze the dependency tree then conditionally trigger CI workflows for affected projects. I wish there was a better way to access the uv dependency worktree other than parsing the `tree` like output

I agree! I hope uv introduces more tools for monorepos or refines the workspaces concept.

I saw workspaces require all dependencies to agree with eachother, which isn't quite possible in our repo

Re: Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

#5

Post by cheesyFish »

tuanacelik wrote: post #4

So just to let me get this straight: Does this new setup aim to make it easier to contribute to llamaindex submodules specifically?

Yes! For example, previously with pants, users would hit a lot of weird errors since how tests run with pants is different than running tests locally with pytest

We did not expect users to learn pants, but this often meant a lot of back and forth with maintainers to get PR tests working.

Should be much easier now!

Re: Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

#6

Post by esafak »

lyjackal wrote: post #2

I recently did something similar. Using uv workspaces, I used the uv CLI's dependency graph to analyze the dependency tree then conditionally trigger CI workflows for affected projects. I wish there was a better way to access the uv dependency worktree other than parsing the `tree` like output

I use Github Actions triggers to pass flags to a monorepo dagger script to build and test the affected components. For example, if a commit touches the front- and back ends, rebuild both. If it only touches the front end, run integration tests using the latest backend without rebuilding it.

edit: spell out GHA

Re: Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

#7

Post by cheesyFish »

esafak wrote: post #6
lyjackal wrote: post #2

I recently did something similar. Using uv workspaces, I used the uv CLI's dependency graph to analyze the dependency tree then conditionally trigger CI workflows for affected projects. I wish there was a better way to access the uv dependency worktree other than parsing the `tree` like output

I use Github Actions triggers to pass flags to a monorepo dagger script to build and test the affected components. For example, if a commit touches the front- and back ends, rebuild both. If it only touches the front end, run integration tests using the latest backend without rebuilding it. edit: spell out GHA

Yea this definitely makes sense for smaller monorepos. For us, we ended up writing our own dependency graph parser to figure out what tests to run (which is easy enough with a single language like python honestly)

Re: Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

#8

Post by esafak »

cheesyFish wrote: post #7
esafak wrote: post #6

Earlier quoted context omitted.

I use Github Actions triggers to pass flags to a monorepo dagger script to build and test the affected components. For example, if a commit touches the front- and back ends, rebuild both. If it only touches the front end, run integration tests using the latest backend without rebuilding it. edit: spell out GHA

Yea this definitely makes sense for smaller monorepos. For us, we ended up writing our own dependency graph parser to figure out what tests to run (which is easy enough with a single language like python honestly)

Was bazel an option?

Re: Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

#9

Post by cheesyFish »

esafak wrote: post #8
cheesyFish wrote: post #7

Earlier quoted context omitted.

Yea this definitely makes sense for smaller monorepos. For us, we ended up writing our own dependency graph parser to figure out what tests to run (which is easy enough with a single language like python honestly)

Was bazel an option?

We used pants initially (which I believe is similar to bazel). And indeed the dependency graphing it does was very helpful, but other parts of the tool motivated us to create something more bespoke and debuggable (we were only using like 20% or less of the features pants offers)
Post reply on HN