Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude is good at assembling blocks, but still falls apart at creating them

approachwithalacrity.com

Post reply on HN
Print view View original

51–60 of 249 posts

Re: Claude is good at assembling blocks, but still falls apart at creating them

#51

Post by ako »

bblcla wrote: post #33
simonw wrote: post #30

I'm not entirely convinced by the anecdote here where Claude wrote "bad" React code: > But in context, this was obviously insane. I knew that key and id came from the same upstream source. So the correct solution was to have the upstream source also pass id to the code that had key, to let it do a fast lookup. I've seen Claude make mistakes like that too, but then the moment you say "you can modify the calling code a…

(Author here) > I'm not entirely convinced by the anecdote here where Claude wrote "bad" React code Yeah, that's fair - a friend of mine also called this out on Twitter ( https://x.com/konstiwohlwend/status/2010799158261936281 ) and I went into more technical detail about the specific problem there. > I've seen Claude make mistakes like that too, but then the moment you say "you can modify the calling code as well" o…

> I agree, but I think I'm less optimistic than you that Claude will be able to catch its own mistakes in the future. On the other hand, I can definitely see how a ~more intelligent model might be able to catch mistakes on a larger and larger scale.

Claude already does this. Yesterday i asked it why some functionality was slow, it did some research, and then came back with all the right performance numbers, how often certain code was called, and opportunities to cache results to speed up execution. It refactored the code, ran performance tests, and reported the performance improvements.

Re: Claude is good at assembling blocks, but still falls apart at creating them

#52

Post by fennecbutt »

maxilevi wrote: post #8

LLMs are just really good search. Ask it to create something and it's searching within the pretrained weights. Ask it to find something and it's semantically searching within your codebase. Ask it to modify something and it will do both. Once you understand its just search, you can get really good results.

I agree somewhat, but more when it comes to its use of logic - it only gleans logic from human language which as we know is a fucking mess.

I've commented before on my belief that the majority of human activity is derivative. If you ask someone to think of a new kind of animal, alien or random object they will always base it off things that they have seen before. Truly original thoughts and things in this world are an absolute rarity and the majority of supposed original thought riffs on what we see others make, and those people look to nature and the natural world for inspiration.

We're very good at taking thing a and thing b and slapping them together and announcing we've made something new. Someone please reply with a wholly original concept. I had the same issue recently when trying to build a magic based physics system for a game I was thinking of prototyping.

Re: Claude is good at assembling blocks, but still falls apart at creating them

#53

Post by fennecbutt »

Scrapemist wrote: post #49

Eventually you can show Claude how you solve problems, and explain the thought process behind it. It can apply these learnings but it will encounter new challenges in doing so. It would be nice if Claude could instigate a conversation to go over the issues in depth. Now it wants quick confirmation to plough ahead.

Well I feel like this is because a better system would distill such learning into tokens not associated with a human language and that that could represent logic better than using English etc for it.

I don't have the GPUs or time to experiment though :(

Re: Claude is good at assembling blocks, but still falls apart at creating them

#54

Post by cultureulterior »

maxilevi wrote: post #8

LLMs are just really good search. Ask it to create something and it's searching within the pretrained weights. Ask it to find something and it's semantically searching within your codebase. Ask it to modify something and it will do both. Once you understand its just search, you can get really good results.

This is not true.

Re: Claude is good at assembling blocks, but still falls apart at creating them

#55

Post by chapel »

bblcla wrote: post #33
simonw wrote: post #30

I'm not entirely convinced by the anecdote here where Claude wrote "bad" React code: > But in context, this was obviously insane. I knew that key and id came from the same upstream source. So the correct solution was to have the upstream source also pass id to the code that had key, to let it do a fast lookup. I've seen Claude make mistakes like that too, but then the moment you say "you can modify the calling code a…

(Author here) > I'm not entirely convinced by the anecdote here where Claude wrote "bad" React code Yeah, that's fair - a friend of mine also called this out on Twitter ( https://x.com/konstiwohlwend/status/2010799158261936281 ) and I went into more technical detail about the specific problem there. > I've seen Claude make mistakes like that too, but then the moment you say "you can modify the calling code as well" o…

Those Python issues are things I had to deal with earlier last year with Claude Sonnet 3.7, 4.0, and to a lesser extent Opus 4.0 when it was available in Claude Code.

In the Python projects I've been using Opus 4.5 with, it hasn't been showing those issues as often, but then again the projects are throwaway and I cared more about the output than the code itself.

The nice thing about these agentic tools is that if you setup feedback loops for them, they tend to fix issues that are brought up. So much of what you bring up can be caught by linting.

The biggest unlock for me with these tools is not letting the context get bloated, not using compaction, and focusing on small chunks of work and clearing the context before working on something else.

Re: Claude is good at assembling blocks, but still falls apart at creating them

#56

Post by XenophileJKO »

andy99 wrote: post #48
bhadass wrote: post #16

Earlier quoted context omitted.

better mental model: it's a lossy compression of human knowledge that can decompress and recombine in novel (sometimes useful, sometimes sloppy) ways. classical search simply retrieves, llms can synthesize as well.

No, this describes the common understanding of LLMs and adds little to just calling it AI. The search is the more accurate model when considering their actual capabilities and understanding weaknesses. “Lossy compression of human knowledge” is marketing.

It is fundamentally and provably different than search because it captures things on two dimensions that can be used combinatorially to infer desired behavior for unobserved examples.

1. Conceptual Distillation - Proven by research work that we can find weights that capture/influence outputs that align with higher level concepts.

2. Conceptual Relations - The internal relationships capture how these concepts are related to each other.

This is how the model can perform acts and infer information way outside of it's training data. Because if the details map to concepts then the conceptual relations can be used to infer desirable output.

(The conceptual distillation also appears to include meta-cognitive behavior, as evidenced by Anthropic's research. Which manes sense to me, what is the most efficient way to be able to replicate irony and humor for an arbitrary subject? Compressing some spectrum of meta-cognitive behavior...)

Re: Claude is good at assembling blocks, but still falls apart at creating them

#57

Post by joshcsimmons »

IDK I've been using opus 4.5 to create a UI library and it's been doing pretty well: https://simsies.xyz/ (still early days)

Granted it was building ontop of tailwind (shifting over to radix after the layoff news). Begs the question? What is a lego?

Re: Claude is good at assembling blocks, but still falls apart at creating them

#58

Post by michalsustr »

bblcla wrote: post #44
doug_durham wrote: post #42

Did the author ask it to make new abstractions? In my experience when I produces output that I don't like I ask it to refactor it. These models have and understanding of all modern design patterns. Just ask it to adopt one.

(Author here) I have! I agree it's very good at applying abstractions, if you know exactly what you want. What I notice is that Claude has almost no ability to surface those abstractions on its own. When I started having it write React, Claude produced incredibly buggy spaghetti code. I had to spend 3 weeks learning the fundamentals of React (how to use hooks, providers, stores, etc.) before I knew how to prompt it t…

Same experience here! As an analogy, consider the model knows both about arabic or roman number representations. But in alternate universe, it has been trained so much on roman numbers ("Bad Code") that it won't give you the arabic ones ("Good Code") unless you prompt it directly, even when they are clearly superior.

I also believe that overall repository code quality is important for AI agents - the more "beautiful" it is, the more the agent can mimic the "beauty".

Re: Claude is good at assembling blocks, but still falls apart at creating them

#59

Post by bblcla »

chapel wrote: post #55
bblcla wrote: post #33

Earlier quoted context omitted.

(Author here) > I'm not entirely convinced by the anecdote here where Claude wrote "bad" React code Yeah, that's fair - a friend of mine also called this out on Twitter ( https://x.com/konstiwohlwend/status/2010799158261936281 ) and I went into more technical detail about the specific problem there. > I've seen Claude make mistakes like that too, but then the moment you say "you can modify the calling code as well" o…

Those Python issues are things I had to deal with earlier last year with Claude Sonnet 3.7, 4.0, and to a lesser extent Opus 4.0 when it was available in Claude Code. In the Python projects I've been using Opus 4.5 with, it hasn't been showing those issues as often, but then again the projects are throwaway and I cared more about the output than the code itself. The nice thing about these agentic tools is that if you…

Arguably linting is a kind of abstraction block!

Re: Claude is good at assembling blocks, but still falls apart at creating them

#60

Post by threethirtytwo »

joshcsimmons wrote: post #57

IDK I've been using opus 4.5 to create a UI library and it's been doing pretty well: https://simsies.xyz/ (still early days) Granted it was building ontop of tailwind (shifting over to radix after the layoff news). Begs the question? What is a lego?

I don't know how someone can look at what you build and conclude LLMs are still google search. It boggles the mind how much hatred people have for AI to the point of self deception. The evidence is placed right in front of you and on your lap with that link and people still deny it.
Post reply on HN