Tips for better coding with ChatGPT

Great tooling.

I was using CodeGPT but this looks better. And it's exciting to have those command line options to easily insert other contextual input like command outputs.

Can I ask about your plans? CodeGPT planned to add semantic embeddings of code, but I haven't seen any updates for quite a time. The idea being that, like walking an Abstract Syntax Tree of the code, you could include relevant functions and function definitions(/classes/constants/etc), to help improve output quality.

Alongside what you note with module/library versions and pasting in doc snippets, I have often found is necessary to include package.json files to nudge the results towards the right API calls. I think semantic code search might push it finally to being able to work even on comprehensive, multi-service style repos.

For MVC or services architected solutions, I think this could be more helpful than including whole files - though it may be overkill - do you have any thoughts on doing this?

>Break down a big change and ask for a series of smaller, self contained steps.

These are all good tips and very much align with my experience over the months, but breaking a complex task into as small, self-contained and logical steps as possible is truly going to make the biggest difference for most people.

Once I understood that, just like previous versions (GPT-3), ChatGPT has no internal memory beyond what has been typed, that changed how I interacted with the model for the better. Depending on the task, I am either providing simple steps or asking the model to write simple steps based on my task layout, then I provide input on those, ask for refinement of said steps with a focus on code generation and remove anything superfluous/not focused on code creation, the output quality became significantly more consistent and usable.

I can understand why, for a lot of more experienced developers, this can seem tedious and fully get why this leads a lot of observers to feel that, considering the effort required to set these models up for success, they might as well code themselves, especially considering this setup process can eat into the precious 25 prompts limit, which I hit consistently.

However, I feel that once these models have become more efficient, a lot of this outlining may be handled in the background.

The same goes for checking code errors in my eyes. For the same reasons (no internal memory), there often can be errors in provided code, yet asking for the model to check for errors generally resolves those. In most cases, this works without providing compile errors, though those of course do improve the response.

I try my hardest not to anthropomorphize these models, so pardon this comparison, but in fairness, even with our human wetware memory the average dev is rarely able to write flawless code without the need for any revisions on their first try.

If we get a more efficient model of comparable quality to GPT-4, adding a line to the frontend to request a second pass on all code based request may not be unreasonable and could, in my experience, yield more consistently usable results.

Considering how quickly OpenAI released gpt-3.5-turbo, I am hopeful that we will see such a development soon, though I currently do not have personal access to the GPT-4 API, so maybe that could be made more accessible first.

Thanks for checking out aider. Let me know if you give it a try and find it useful.

Yes, aider already has features to provide GPT with "code context" to let it edit larger, more complex codebases. I wrote up some notes about these features:

https://aider.chat/docs/ctags.html

You might be especially interested in the "future work" section near the end. I have actually shipped some of these ideas into the tool already. I need to find some time to write up the details.

Certainly, but GPT3.5 is effectively free via the API, especially considering you have to hit $5 or whatever to even be as expensive as ChatGPT (and it's even relatively difficult to rack up a high bill on GPT4 now that they have token rate limits). The problem of general and global access to unrestricted LLMs is a very, very serious question, but here it's largely a question of engineers and, since it's relatively difficult to spend more than a Netflix subscription via the API, I'm assuming this is conceivably in budget.

> When these things can understand the business requirements

I’ve been looking into this. Nothing definitive yet, but my hunch is that current LLM’s struggle because they lack curiosity. They will answer your question, however vague, with the first most obvious answer they can think of.

This is great, if you’re a junior team member. Super talented, very eager. But a more senior engineer approaches the problem differently. They ask more questions than provide answers. They’ll happily spend the first 25min of a 30min meeting asking heaps upon oodles of really dumb sounding questions.

Then the last 5min, that’s the magic. They now have a solution perfectly tailored to the problem at hand, with all sorts of edge cases either explored or eliminated through questioning. The questions they kept asking weren’t dumb after all, they were pruning a decision/option tree in their head of all possible solutions until they landed on the most optimal solution for a set of known constraints. With further options to dig and improve.

I think you can make an LLM do this (i’m trying), but it’s very very slow still.