Show HN: GPT Repo Loader – load entire code repos into GPT prompts • Hacker News Board

Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#1

Post by mpoon » Fri, Mar 17, 2023, 12:59 AM UTC

I was getting tired of copy/pasting reams of code into GPT-4 to give it context before I asked it to help me, so I started this small tool. In a nutshell, gpt-repository-loader will spit out file paths and file contents in a prompt-friendly format. You can also use .gptignore to ignore files/folders that are irrelevant to your prompt.

gpt-repository-loader as-is works pretty well in helping me achieve better responses. Eventually, I thought it would be cute to load itself into GPT-4 and have GPT-4 improve it. I was honestly surprised by PR#17. GPT-4 was able to write a valid an example repo and an expected output and throw in a small curveball by adjusting .gptignore. I did tell GPT the output file format in two places: 1.) in the preamble when I prompted it to make a PR for issue #16 and 2.) as a string in gpt_repository_loader.py, both of which are indirect ways to infer how to build a functional test. However, I don't think I explained to GPT in English anywhere on how .gptignore works at all!

I wonder how far GPT-4 can take this repo. Here is the process I'm following for developing:

- Open an issue describing the improvement to make

- Construct a prompt - start with using gpt_repository_loader.py on this repo to generate the repository context, then append the text of the opened issue after the --END-- line.

- Try not to edit any code GPT-4 generates. If there is something wrong, continue to prompt GPT to fix whatever it is.

- Create a feature branch on the issue and create a pull request based on GPT's response.

- Have a maintainer review, approve, and merge.

I am going to try to automate the steps above as much as possible. Really curious how tight the feedback loop will eventually get before something breaks!

Show HN: GPT Repo Loader – load entire code repos into GPT prompts
github.com

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#2

Post by bebrws » Fri, Mar 17, 2023, 1:18 AM UTC

From what I understand this seems useful if you have a model that will accept a large or unlimited number of tokens. I was looking into doing the same thing with ChatGPT and went with ada to find snippets related to the prompt and then to include those with a prompt to ChatGPT: https://bbarrows.com/posts/using-embeddings-ada-and-chatgpt-...

Does ChatGPT 4 now accept more tokens maybe?

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#3

Post by EGreg » Fri, Mar 17, 2023, 1:18 AM UTC

How much text can you feed GPT-4?

Our codebase is 1 million lines of code.

Can we feed the documentation to it? What are the limits?

Is it possible to train it on our data without doing prompt engineering? How?

Otherwise are we supposed to use embeddings? Can someone explain how these all work and the tradeoffs?

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#4

Post by blurbleblurble » Fri, Mar 17, 2023, 1:22 AM UTC

EGreg wrote: post #3

How much text can you feed GPT-4? Our codebase is 1 million lines of code. Can we feed the documentation to it? What are the limits? Is it possible to train it on our data without doing prompt engineering? How? Otherwise are we supposed to use embeddings? Can someone explain how these all work and the tradeoffs?

32k tokens is the limit, so you won't be able to load the whole thing into the context.

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#5

Post by DaiPlusPlus » Fri, Mar 17, 2023, 1:22 AM UTC

Rather than prompting GPT into implementing a solution, can we prompt it to try to preemptively find issues with the codebase or missing-functionality?

Also, do we know what languages GPT-4 "understands" at a sufficient level? What knowledge does it have of post-2021 language features, like in C23?

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#6

Post by textninja » Fri, Mar 17, 2023, 1:24 AM UTC

I’m curious to see how this turns out as well, though you’ll probably have to devise a workaround for the token limit for this to be effective for all but the smallest projects.

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#7

Post by mpoon » Fri, Mar 17, 2023, 1:24 AM UTC

EGreg wrote: post #3

How much text can you feed GPT-4? Our codebase is 1 million lines of code. Can we feed the documentation to it? What are the limits? Is it possible to train it on our data without doing prompt engineering? How? Otherwise are we supposed to use embeddings? Can someone explain how these all work and the tradeoffs?

I'm waiting on my GPT-4 API access so I can use gpt-4-32k which maybe can soak up 10k LOC?

Clearly this will break eventually, but I am playing around with some ideas to extend how much context I can give it. One is to do something like base64 encode file contents. I've seen some early success that GPT-4 knows how to decode it, so that'll allow me to stuff more characters into it. I'm also hoping that with the use of .gptignore, I can just selectively give the files I think are relevant for whatever prompt I'm writing.

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#8

Post by ftufek » Fri, Mar 17, 2023, 1:30 AM UTC

DaiPlusPlus wrote: post #5

Rather than prompting GPT into implementing a solution, can we prompt it to try to preemptively find issues with the codebase or missing-functionality? Also, do we know what languages GPT-4 "understands" at a sufficient level? What knowledge does it have of post-2021 language features, like in C23?

It has no post-2021 knowledge, but while playing with it, I found that you can just paste the documentation (no need to even format it) and it'll just "learn" it. For example, safetensors wasn't available back then apparently, I just copied the docs into it and was able to get it write pretty good pytorch code that incorporates safetensors.

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#9

Post by textninja » Fri, Mar 17, 2023, 1:31 AM UTC

DaiPlusPlus wrote: post #5

Rather than prompting GPT into implementing a solution, can we prompt it to try to preemptively find issues with the codebase or missing-functionality? Also, do we know what languages GPT-4 "understands" at a sufficient level? What knowledge does it have of post-2021 language features, like in C23?

I imagine few shot learning would kick in for most new language features. A feature may be new to a particular language, but is it really new?

Re: Show HN: GPT Repo Loader – load entire code repos into GPT prompts

#10

Post by ftufek » Fri, Mar 17, 2023, 1:34 AM UTC

This is awesome, can't wait to get api access to the 32k token model. Rather than this approach of just converting the whole repo to a text file, what I'm thinking is, you can let the model decide the most relevant files.

The initial prompt would be, "person wants to do x, here are the file list of this repo: ...., give me a list of files that you'd want to edit, create or delete" -> take the list, try to fit the contents of them into 32k tokens and re-prompt with "user is trying to achieve x, here's the most relevant files with their contents:..., give me a git commit in the style of git patch/diff output". From playing around with it today, I think this approach would work rather well and can be like a huge step up from AI line autocompletion.