Live data from Hacker News

Scaling LLMs to Larger Codebases

blog.kierangill.xyz

91–100 of 123 posts

Re: Scaling LLMs to Larger Codebases

#91
post #41

I have a somewhat different take on this (somewhat captured in the post linked below). IMO, the best way to raise the floor of LLM performance in codebases is by building meaning into the code base itself ala DDD. If your codebase is hard to understand and grok for a human, it will be the same for an LLM. If your codebase is unstructured and has no definable patterns, it will be harder for an LLM to use. You can try…

Great post. I work on two large codebases. One is structured much like the example from the post, and the other is a mess. LLMs care much better at understanding the organized code.

Re: Scaling LLMs to Larger Codebases

#92
post #46

LLMs are so good at telling me about things I know little to nothing about, but when when I ask about things I have expert knowledge on they consistently fail, hallucinate, and confidently lie...

You just don't know enough to identify the bullshit when you aren't an expert in that domain.

Re: Scaling LLMs to Larger Codebases

#93
There is no way this is economical.

Burn through your token limit in agent mode just to thrash around a few more times trying to identify where the agent "misunderstood" the prompt.

The only time LLM's work as coding agents for me is tightly scoped prompts with a small isolated context.

Just throwing an entire codebase into an LLM in an agentic loop seems like a fools errand.

Re: Scaling LLMs to Larger Codebases

#94
If you're interested in the large codebase... The best I found so far are extended context models. Using newest Nemotron3 nano, you can put a 1m tokens (about 3 ish megabytes of text) of pure code dump (I use repomix --style markdown) and ask around. That's been one of the biggest wow moments I had with LLMs so far. Much better experience than any RAG I used

Re: Scaling LLMs to Larger Codebases

#95
post #4

> Making a prompt library useful requires iteration. Every time the LLM is slightly off target, ask yourself, "What could've been clarified?" Then, add that answer back into the prompt library. I'm far from an LLM power user, but this is the single highest ROI practice I've been using. You have to actually observe what the LLM is trying to do each time. Simply smashing enter over and over again or setting it to auto-…

> Every time the LLM is slightly off target, ask yourself, "What could've been clarified?

Better than that, ask the LLM. Better than that, have the LLM ask itself. You do still have make sure it doesn't go off the rails, but the LLM itself wrote this to help answer the question:

### Pattern 10: Student Pattern (Fresh Eyes)

*Concept:* Have a sub-agent read documentation/code/prompts "as a newcomer" to find gaps, contradictions, and confusion points that experts miss.

*Why it works:* Developers write with implicit knowledge they don't realize is missing. A "student" perspective catches assumptions, undefined terms, and inconsistencies.

*Example prompt:* ``` Task: "Student Pattern Review

Pretend you are a NEW AI agent who has never seen this codebase. Read these docs as if encountering them for the first time: 1. CLAUDE.md 2. SUB_AGENT_QUICK_START.md

Then answer from a fresh perspective:

## Confusion Points - What was confusing or unclear on first read? - What terms are used without explanation?

## Contradictions - Where do docs disagree with each other? - What's inconsistent?

## Missing Information - What would a new agent need to know that isn't covered?

## Recommendations - Concrete edits to improve clarity

Be honest and critical. Include file:line references." ```

*Uses cases:* Before finalizing new documentation, evaluating prompts for future Agents.

Re: Scaling LLMs to Larger Codebases

#96
One thing that helped us as codebases grew was separating decision-making from execution. Let the model reason about intent and scope, but keep execution deterministic and constrained. It reduced drift and made failures much easier to debug once context got large.

Re: Scaling LLMs to Larger Codebases

#97
“When an LLM can generate a working high-quality implementation in a single try, that is called one-shotting. This is the most efficient form of LLM programming.”

This is a good article, but misses one of the most important advances this year - the agentic loop.

There are always going to be limits to how much code a model can one-shot. Give it the ability to verify its changes and iterate, massively increase its ability to write sizeable chunks of verified and working code.

Re: Scaling LLMs to Larger Codebases

#98
post #64
post #62

Earlier quoted context omitted.

I’m uneasy having an agent implement several pages of plan and then writing tests and results only at the and of all that. It feels like getting a CS student to write and follow a plan to do something they haven’t worked on before. It’ll report, “Numbers changed in step 6a therefore it worked” [forgetting the pivotal role of step 2 which failed and as a result the agent should have taken step 6b, not 6a]. Or “there i…

This is a bit like agile versus waterfall.

100%, the reason I thought of this is constantly telling developers to break their work down into smaller pieces so that they can focus and the customer sees value sooner.

Re: Scaling LLMs to Larger Codebases

#99
post #46

LLMs are so good at telling me about things I know little to nothing about, but when when I ask about things I have expert knowledge on they consistently fail, hallucinate, and confidently lie...

I’ve found that they vary a huge amount based on the subject matter. In my case, I have noticed the opposite of what you observed. They know a lot about the web space (which I’ve been in for around 25 years), but are pretty bad (though not useless) at esoteric languages such as Hare.

Obviously, since the training material for such esoteric languages is scarce. (That's why they are esoteric!) So by definition, LLM will never be good at esoteric languages.

Re: Scaling LLMs to Larger Codebases

#100
Put every detail into CLAUDE.md and after a while CC starts to forget/ignore what it’s been told.

I’d like to see dynamic task-specific context building. Write a prompt and the model starts to collect relevant instructions.

Also a review loop to check that instructions were followed.

Post reply on HN