Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The highest quality codebase

gricha.dev

Post reply on HN
Print view View original

21–30 of 409 posts

Re: The highest quality codebase

#21

Post by postalcoder »

One of my favorite personal evals for llms is testing its stability as a reviewer.

The basic gist of it is to give the llm some code to review and have it assign a grade multiple times. How much variance is there in the grade?

Then, prompt the same llm to be a "critical" reviewer with the same code multiple times. How much does that average critical grade change?

A low variance of grades across many generations and a low delta between "review this code" and "review this code with a critical eye" is a major positive signal for quality.

I've found that gpt-5.1 produces remarkably stable evaluations whereas Claude is all over the place. Furthermore, Claude will completely [and comically] change the tenor of its evaluation when asked to be critical whereas gpt-5.1 is directionally the same while tightening the screws.

You could also interpret these results to be a proxy for obsequiousness.

Edit: One major part of the eval i left out is "can an llm converge on an 'A'?" Let's say the llm gives the code a 6/10 (or B-). When you implement its suggestions and then provide the improved code in a new context, does the grade go up? Furthermore, can it eventually give itself an A, and consistently?

It's honestly impressive how good, stable, and convergent gpt-5.1 is. Claude is not great. I have yet to test it on Gemini 3.

Re: The highest quality codebase

#22

Post by rtp4me »

kderbyma wrote: post #3

Yeah. I noticed Claud suffers when it reaches context overload - its too opinionated, so it shortens its own context with decisions I would not ever make, yet I see it telling itself that the shortcuts are a good idea because the project is complex...then it gets into a loop where it second guesses its own decisions and forgets the context and then continues to spiral uncontrollably into deeper and deeper failures -…

For me, too many compactions throughout the day eventually lead to a decline in Claude's thinking ability. And, during that time, I have given it so much context to help drive the coding interaction. Thus, restarting Claude requires me to remember the small bits of "nuggets" we discovered during the last session so I find myself repeating the same things every day (my server IP is: xxx, my client IP is: yyy, the code should live in directory: a/b/c). Using the resume feature with Claude simply brings back the same decline in thinking that led me to stop it in the first place. I am sure there is a better way to remember these nuggets between sessions but I have not found it yet.

Re: The highest quality codebase

#24

Post by dcchuck »

I spent some time last night "over iterating" on a plan to do some refactoring in a large codebase.

I created the original plan with a very specific ask - create an abstraction to remove some tight coupling. Small problem that had a big surface area. The planning/brainstorming was great and I like the plan we came up with.

I then tried to use a prompt like OP's to improve it (as I said, large surface area so I wanted to review it) - "Please review PLAN_DOC.md - is it a comprehensive plan for this project?". I'd run it -> get feedback -> give it back to Claude to improve the plan.

I (naively perhaps) expected this process to converge to a "perfect plan". At this point I think of it more like a probability tree where there's a chance of improving the plan, but a non-zero chance of getting off the rails. And once you go off the rails, you only veer further and further from the truth.

There are certainly problems where "throwing compute" at it and continuing to iterate with an LLM will work great. I would expect those to have firm success criteria. Providing definitions of quality would significantly improve the output here as well (or decrease the probability of going off the rails I suppose). Otherwise Claude will confuse quality like we see here.

Shout out OP for sharing their work and moving us forward.

Re: The highest quality codebase

#25

Post by plufz »

xnorswap wrote: post #18

Claude is really good at specific analysis, but really terrible at open-ended problems. "Hey claude, I get this error message: ", and it'll often find the root cause quicker than I could. "Hey claude, anything I could do to improve Y?", and it'll struggle beyond the basics that a linter might suggest. It suggested enthusiastically a library for and it was all " Recommended " about it, but when I pointed out that the…

I think slash commands are great to help Claude with this. I have many like /code:dry /code:clean-code etc that has a semi long prompt and references to longer docs to review code from a specific perspective. I think it atleast improves Claude a bit in this area. Like processes or templates for thinking in broader ways. But yes I agree it struggles a lot in this area.

Re: The highest quality codebase

#26

Post by kccqzy »

xnorswap wrote: post #18

Claude is really good at specific analysis, but really terrible at open-ended problems. "Hey claude, I get this error message: ", and it'll often find the root cause quicker than I could. "Hey claude, anything I could do to improve Y?", and it'll struggle beyond the basics that a linter might suggest. It suggested enthusiastically a library for and it was all " Recommended " about it, but when I pointed out that the…

Not at all my experience. I’ve often tried things like telling Claude this SIMD code I wrote performed poorly and I needed some ideas to make it go faster. Claude usually does a good job rewriting the SIMD to use different and faster operations.

Re: The highest quality codebase

#27

Post by ebonnafoux »

xixixao wrote: post #11
m101 wrote: post #4

This is a great example of there being no intelligence under the hood.

Would a human perform very differently? A human who must obey orders (like maybe they are paid to follow the prompt). With some "magnitude of work" enforced at each step. I'm not sure there's much to learn here, besides it's kinda fun, since no real human was forced to suffer through this exercise on the implementor side.

I have seen some codebase doubling the number of LoC after "refactoring" made by humans, so I would say no.

Re: The highest quality codebase

#28

Post by turboponyy »

written-beyond wrote: post #2

> I like Rust's result-handling system, I don't think it works very well if you try to bring it to the entire ecosystem that already is standardized on error throwing. I disagree, it's very useful even in languages that have exception throwing conventions. It's good enough for the return type for Promise.allSettled api. The problem is when I don't have the result type I end up approximating it anyway through other wa…

I have the same disagreement. TypeScript with its structural and pseudo-dependent typing, somewhat-functionally disposed language primitives (e.g. first-class functions as values, currying) and standard library interfaces (filter, reduce, flatMap et al), and ecosystem make propagating information using values extremely ergonomic.

Embracing a functional style in TypeScript is probably the most productive I've felt in any mainstream programming language. It's a shame that the language was defiled with try/catch, classes and other unnecessary cruft so third party libraries are still an annoying boundary you have to worry about, but oh well.

The language is so well-suited for this that you can even model side effects as values, do away with try/catch, if/else and mutation a la Haskell, if you want[1].

[1] https://effect.website/

Re: The highest quality codebase

#29

Post by james_marks »

xnorswap wrote: post #18

Claude is really good at specific analysis, but really terrible at open-ended problems. "Hey claude, I get this error message: ", and it'll often find the root cause quicker than I could. "Hey claude, anything I could do to improve Y?", and it'll struggle beyond the basics that a linter might suggest. It suggested enthusiastically a library for and it was all " Recommended " about it, but when I pointed out that the…

This is a key part of the AI love/hate flame war.

Very easy to write it off when it spins out on the open-ended problems, without seeing just how effective it can be once you zoom in.

Of course, zooming in that far gives back some of the promised gains.

Edit: typo

Re: The highest quality codebase

#30

Post by pawelduda »

f311a wrote: post #10

I like to ask LLMs to find problems o improvements in 1-2 files. They are pretty good at finding bugs, but for general code improvements, 50-60% edits are trash. They add completely unnecessary stuff. If you ask them to improve a pretty well-written code, they rarely say it's good enough already. For example, in a functional-style codebase, they will try to rewrite everything to a class. I have to adjust the prompt t…

If you just ask it to find problems, it will do its best to find them - like running a while loop with no return condition. That's why I put some breaker in the prompt, which in this case would be "don't make any improvements if the positive impact is marginal". I've mostly seen it do nothing and just summarize why, followed by some suggestions in case I still want to force the issue
Post reply on HN