I have a somewhat different take on this (somewhat captured in the post linked below). IMO, the best way to raise the floor of LLM performance in codebases is by building meaning into the code base itself ala DDD. If your codebase is hard to understand and grok for a human, it will be the same for an LLM. If your codebase is unstructured and has no definable patterns, it will be harder for an LLM to use. You can try…
Scaling LLMs to Larger Codebases
91–100 of 123 posts
Re: Scaling LLMs to Larger Codebases
#92LLMs are so good at telling me about things I know little to nothing about, but when when I ask about things I have expert knowledge on they consistently fail, hallucinate, and confidently lie...
Re: Scaling LLMs to Larger Codebases
#93Burn through your token limit in agent mode just to thrash around a few more times trying to identify where the agent "misunderstood" the prompt.
The only time LLM's work as coding agents for me is tightly scoped prompts with a small isolated context.
Just throwing an entire codebase into an LLM in an agentic loop seems like a fools errand.
Re: Scaling LLMs to Larger Codebases
#94Re: Scaling LLMs to Larger Codebases
#95> Making a prompt library useful requires iteration. Every time the LLM is slightly off target, ask yourself, "What could've been clarified?" Then, add that answer back into the prompt library. I'm far from an LLM power user, but this is the single highest ROI practice I've been using. You have to actually observe what the LLM is trying to do each time. Simply smashing enter over and over again or setting it to auto-…
Better than that, ask the LLM. Better than that, have the LLM ask itself. You do still have make sure it doesn't go off the rails, but the LLM itself wrote this to help answer the question:
### Pattern 10: Student Pattern (Fresh Eyes)
*Concept:* Have a sub-agent read documentation/code/prompts "as a newcomer" to find gaps, contradictions, and confusion points that experts miss.
*Why it works:* Developers write with implicit knowledge they don't realize is missing. A "student" perspective catches assumptions, undefined terms, and inconsistencies.
*Example prompt:* ``` Task: "Student Pattern Review
Pretend you are a NEW AI agent who has never seen this codebase. Read these docs as if encountering them for the first time: 1. CLAUDE.md 2. SUB_AGENT_QUICK_START.md
Then answer from a fresh perspective:
## Confusion Points - What was confusing or unclear on first read? - What terms are used without explanation?
## Contradictions - Where do docs disagree with each other? - What's inconsistent?
## Missing Information - What would a new agent need to know that isn't covered?
## Recommendations - Concrete edits to improve clarity
Be honest and critical. Include file:line references." ```
*Uses cases:* Before finalizing new documentation, evaluating prompts for future Agents.
Re: Scaling LLMs to Larger Codebases
#96Re: Scaling LLMs to Larger Codebases
#97This is a good article, but misses one of the most important advances this year - the agentic loop.
There are always going to be limits to how much code a model can one-shot. Give it the ability to verify its changes and iterate, massively increase its ability to write sizeable chunks of verified and working code.
Re: Scaling LLMs to Larger Codebases
#98Earlier quoted context omitted.
I’m uneasy having an agent implement several pages of plan and then writing tests and results only at the and of all that. It feels like getting a CS student to write and follow a plan to do something they haven’t worked on before. It’ll report, “Numbers changed in step 6a therefore it worked” [forgetting the pivotal role of step 2 which failed and as a result the agent should have taken step 6b, not 6a]. Or “there i…
This is a bit like agile versus waterfall.
Re: Scaling LLMs to Larger Codebases
#99LLMs are so good at telling me about things I know little to nothing about, but when when I ask about things I have expert knowledge on they consistently fail, hallucinate, and confidently lie...
I’ve found that they vary a huge amount based on the subject matter. In my case, I have noticed the opposite of what you observed. They know a lot about the web space (which I’ve been in for around 25 years), but are pretty bad (though not useless) at esoteric languages such as Hare.
Re: Scaling LLMs to Larger Codebases
#100I’d like to see dynamic task-specific context building. Write a prompt and the model starts to collect relevant instructions.
Also a review loop to check that instructions were followed.