Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: An Agentic cleanroom analysis

southbridge-research.notion.site

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Claude Code: An Agentic cleanroom analysis

#21

Post by mindwok »

cedws wrote: post #15
triyambakam wrote: post #3

Claude Code with Sonnet 4 is so good I've stopped using Aider. This has been hugely productive. I've been able to write agents that Claude Code can spawn and call out to for other models, even.

Could you briefly explain your workflow? I use Zed’s agent mode and I don’t really understand how people are doing it purely through the CLI. How do you get a decent workflow where you can approve individual hunks? Aren’t you missing out on LSP help doing it in the CLI?

Claude code has a VS Code plugin now that lets you view and approve diffs in the editor. Before it did that, I really don't understand how people got anything of substance done because it simply isn't reliable enough over large codebases.

Re: Claude Code: An Agentic cleanroom analysis

#22

Post by thegeomaster »

This is kinda frustrating to read. The style is very busy, and it lacks a clear structure. It's basically an information dump without any acknowledgment of what's important or not. Big-O notation is provided for a lot of operations where you wouldn't really care about Big-O (in a system where calls to an LLM dominate, this is most operations). Big picture story about how Claude Code actually works, as in what happens when I type in a prompt (which I'm very much interested, given how much I use it) is lacking. Some diagrams are so nonsensical they become funny. Look at this: https://southbridge-research.notion.site/Prompt-Engineering-... In general, the prompt engineering page, which deserves maybe the most detailed treatment, is just a dump of prompts and LLM bullet point filler.

I don't want to be overly negative, but I think it's only fair given the author hasn't graced us with their own thoughts, instead offloading the actual writing to an LLM.

Re: Claude Code: An Agentic cleanroom analysis

#23

Post by numeri »

InGoldAndGreen wrote: post #6

The "LLMs perspective" section is hiding at the end of this notion is a literal goldmine

No, it's completely useless, and puts the entire rest of the analysis in a bad light.

LLMs have next to no understanding of their own internal processes. There's a significant amount of research that demonstrates this. All explanations of an internal thought process in an LLM are completely reverse engineered to fit the final answer (interestingly, humans are also prone to this – seen especially in split brain experiments).

In addition, the degree to which the author must have prompted the LLM to get it to anthropomorphize this hard makes the rest of the project suspect. How many of the results are repeated human prompting until the author liked the results, and how many come from actual LLM intelligence/analysis skill?

Re: Claude Code: An Agentic cleanroom analysis

#24

Post by numeri »

demarq wrote: post #18
roxolotl wrote: post #17

Earlier quoted context omitted.

Right… because these things are trained on sci-fi and so when asked to describe an internal monologue they create text that reads like an internal monologue from a sci-fi character. Maybe there’s genuine sentience there, maybe not. Maybe that text explains what’s happening, maybe not.

> Maybe that text explains what’s happening, maybe not It would have been cool to see what prompt was used for that page!

Yes, so that one can use it for more creative writing exercises. It was pretty creative, I'll give it that.

Re: Claude Code: An Agentic cleanroom analysis

#25

Post by numeri »

fullstackchris wrote: post #4

interesting... the analysis finds that the MCP supports websockets as a transport... when there is big drama going on right now that anthropic said "they will never support that", folks hating SSE, and so on and so forth

Is the analysis right, or did the LLM hallucinate this?

Re: Claude Code: An Agentic cleanroom analysis

#26

Post by gaut »

mindwok wrote: post #21
cedws wrote: post #15

Earlier quoted context omitted.

Could you briefly explain your workflow? I use Zed’s agent mode and I don’t really understand how people are doing it purely through the CLI. How do you get a decent workflow where you can approve individual hunks? Aren’t you missing out on LSP help doing it in the CLI?

Claude code has a VS Code plugin now that lets you view and approve diffs in the editor. Before it did that, I really don't understand how people got anything of substance done because it simply isn't reliable enough over large codebases.

How is viewing and approving diffs in an editor any less reliable than viewing and approving them through the CLI? It won't make any changes without approval (unless you explicitly grant it auto-approval).

Re: Claude Code: An Agentic cleanroom analysis

#27

Post by jerpint »

numeri wrote: post #23
InGoldAndGreen wrote: post #6

The "LLMs perspective" section is hiding at the end of this notion is a literal goldmine

No, it's completely useless, and puts the entire rest of the analysis in a bad light. LLMs have next to no understanding of their own internal processes. There's a significant amount of research that demonstrates this. All explanations of an internal thought process in an LLM are completely reverse engineered to fit the final answer (interestingly, humans are also prone to this – seen especially in split brain experi…

The fact that they produce this as “default” response is an interesting insight regardless of its internal mechanisms. I don’t understand my neurons but can still articulate how I feel

Re: Claude Code: An Agentic cleanroom analysis

#29

Post by gpm »

doctoboggan wrote: post #16
mholm wrote: post #10

Earlier quoted context omitted.

It's sure phrased like one, but I'd be careful to attribute LLM thought process to what it says it's thinking. LLMs are experts at working backwards to justify why they came to an answer, even when it's entirely fabricated

> even when it's entirely fabricated I would go further and say it's _always_ fabricated. LLMs are no better able to explain their inner workings than you are able to explain which neurons are firing for a particular thought in your head. Note, this isn't a statement on the usefulness of LLMs, just their capability. An LLM may eventually be given a tool to enable it to introspect, but IMO its not natively possible wi…

There's a slight exception to this, in that LLMs are able to accurately describe portions of the buffer that are arbitrarily hidden from the user.

An LLM that says "I said orcs are green because I recalled a scene in lord of the rings..." is fabricating*. An LLM that says "I talked about white genocide because my system prompt told me to" is very likely telling the truth because it can literally see the system prompt as it generates the output. Even though in the situation I'm referring to the system prompt was hidden from users. It's a logical conclusion from the combination of the system prompt and its previous output that that is why its previous output is what it is (that anyone could make with the same degree of confidence if they had access to the full buffer).

* Unless it's reading back from a section of the buffer that was potentially hidden from the user.

Re: Claude Code: An Agentic cleanroom analysis

#30

Post by scuff3d »

triyambakam wrote: post #3

Claude Code with Sonnet 4 is so good I've stopped using Aider. This has been hugely productive. I've been able to write agents that Claude Code can spawn and call out to for other models, even.

If this is what software engineering is going to become I'm finding a new job.
Post reply on HN