Show HN: How we leapfrogged traditional vector based RAG with a 'language map' • Hacker News Board

I want to encourage you all to ask the chat some tough questions. You can ask very complex and general questions. Some examples: - Ask ollama (https://wiki.mutable.ai/ollama/ollama) how to add a new model - Ask langchain (https://wiki.mutable.ai/langchain-ai/langchain) "How can I build a simple personal assistant using this repo?" - Ask flash attention (https://wiki.mutable.ai/Dao-AILab/flash-attention) "What are the main benefits of using this code?"

It is also useful for search - for example, if you ask langchain "Where is the code that connects to vector databases?" it will surface all the relevant information.

Very curious to hear what you ask (and whether you find the response helpful)!

Re: Show HN: How we leapfrogged traditional vector based RAG with a 'language map'

#43

Post by oshams » Fri, Jul 19, 2024, 5:49 PM UTC

J_Shelby_J wrote: post #39

This is amazing. Do you have any information on the implementation?

thanks! we tried to include some info in this post.

Re: Show HN: How we leapfrogged traditional vector based RAG with a 'language map'

#44

Post by anotherpaulg » Fri, Jul 19, 2024, 6:03 PM UTC

manishsharan wrote: post #30

anotherpaulg wrote: post #11

I agree that many AI coding tools have rushed to adopt naive RAG on code. Have you done any quantitative evaluation of your wiki style code summaries? My first impression is that they might be too wordy and not deliver valuable context in a token efficient way. Aider uses a repository map [0] to deliver code context. Relevant code is identified using a graph optimization on the repository's AST & call graph, not vect…

I tried using Aider but my codebase is a mix of Clojure Clojurescript and Java . I gave up making it work for me it as it created more issues for me. What I really hated about Aider was that it made code changes without my approval.

The recommended workflow is to just use /undo to revert any edits that you don’t like.

Re: Show HN: How we leapfrogged traditional vector based RAG with a 'language map'

#45

Post by billmalarky » Fri, Jul 19, 2024, 6:20 PM UTC

oshams wrote: post #40

BTW, if you ask for an open source repo in the comments to be processed by our system, we'll do it for free!

Can you share some basic insight into how your system processes an open source repo to generate a wiki with a hierarchy and structure that maps to the same or similar hierarchy and structure of the codebase?

I can understand the value of marrying that wiki to the codebase and how that would help LLM's better "understand" the codebase.

What I'm lost on is how you can auto-generate that wiki in the first place (in a high quality way). I presume it's not perfect, but a very interesting problem space and would love to hear what you've learned and what you are trying to accomplish this feat!

Thanks for posting btw, this HN comments section has been INCREDIBLE.

Re: Show HN: How we leapfrogged traditional vector based RAG with a 'language map'

#46

Post by zellyn » Fri, Jul 19, 2024, 6:28 PM UTC

These wikis are really interesting. I'm itching to try it on the common framework parts of our work monorepo.

[Update after looking at the Django wiki]

The wiki's structure appears to be very (entirely?) based on the directory structure. Is that right?

It would be interesting to give it the Django documentation in addition to the codebase, or possibly even just the heading structure of the documentation, and let it use that as a scaffold to generate the structure/organization of the wiki.

For a specific example, the Model is one of the most important concepts in Django, but in the auto-generated wiki, it shows up underneath databases.

Re: Show HN: How we leapfrogged traditional vector based RAG with a 'language map'

#47

Post by zellyn » Fri, Jul 19, 2024, 6:38 PM UTC

zellyn wrote: post #46

These wikis are really interesting. I'm itching to try it on the common framework parts of our work monorepo. [Update after looking at the Django wiki] The wiki's structure appears to be very (entirely?) based on the directory structure. Is that right? It would be interesting to give it the Django documentation in addition to the codebase, or possibly even just the heading structure of the documentation, and let it u…

Out of curiosity, would you mind pointing it at simonw/datasette? I think that might be interesting, particularly due to the plugin system.

Re: Show HN: How we leapfrogged traditional vector based RAG with a 'language map'

#48

Post by oshams » Fri, Jul 19, 2024, 6:41 PM UTC

billmalarky wrote: post #45

oshams wrote: post #40

BTW, if you ask for an open source repo in the comments to be processed by our system, we'll do it for free!

Can you share some basic insight into how your system processes an open source repo to generate a wiki with a hierarchy and structure that maps to the same or similar hierarchy and structure of the codebase? I can understand the value of marrying that wiki to the codebase and how that would help LLM's better "understand" the codebase. What I'm lost on is how you can auto-generate that wiki in the first place (in a hi…

Thanks! It's a multi stage process where we summarize the code and structure it into articles. We also do some self verification, for example checking for dead links.

Re: Show HN: How we leapfrogged traditional vector based RAG with a 'language map'

#49

Post by billmalarky » Fri, Jul 19, 2024, 6:47 PM UTC

oshams wrote: post #48

billmalarky wrote: post #45

Earlier quoted context omitted.

Can you share some basic insight into how your system processes an open source repo to generate a wiki with a hierarchy and structure that maps to the same or similar hierarchy and structure of the codebase? I can understand the value of marrying that wiki to the codebase and how that would help LLM's better "understand" the codebase. What I'm lost on is how you can auto-generate that wiki in the first place (in a hi…

Thanks! It's a multi stage process where we summarize the code and structure it into articles. We also do some self verification, for example checking for dead links.

Do you build a syntax tree of the code, then loop the tree to auto-write an article for each node (or the larger or more material nodes) and then also reference the tree to pull in related nodes/pieces/modules/whatnot of the codebase when auto-writing a documentation article for each node?

This is fascinating thank you

Re: Show HN: How we leapfrogged traditional vector based RAG with a 'language map'

#50

Post by oshams » Fri, Jul 19, 2024, 6:52 PM UTC

billmalarky wrote: post #49

oshams wrote: post #48

Earlier quoted context omitted.

Thanks! It's a multi stage process where we summarize the code and structure it into articles. We also do some self verification, for example checking for dead links.

Do you build a syntax tree of the code, then loop the tree to auto-write an article for each node (or the larger or more material nodes) and then also reference the tree to pull in related nodes/pieces/modules/whatnot of the codebase when auto-writing a documentation article for each node? This is fascinating thank you

yeah ! you know the best way to learn more is to join us, we're hiring: https://www.workatastartup.com/companies/mutable-ai