Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude is good at assembling blocks, but still falls apart at creating them

approachwithalacrity.com

Post reply on HN
Print view View original

171–180 of 249 posts

Re: Claude is good at assembling blocks, but still falls apart at creating them

#171

Post by skue »

woeirua wrote: post #133

It's just amazing to me how fast the goal posts are moving. Four years ago, if you had told someone that a LLM would be able to one-shot either of those first two tasks they would've said you're crazy. The tech is moving so fast . I slept on Opus 4.5 because GPT 5 was kind of an air ball, and just started using it in the past few weeks. It's so good. Way better than almost anything that's come before it. It can one-s…

> Four years ago, if you had told someone that a LLM would be able to one-shot either of those first two tasks they would've said you're crazy.

Four years ago, they would have likely asked what in the world is an LLM? ChatGPT is barely 3 years old.

Re: Claude is good at assembling blocks, but still falls apart at creating them

#172

Post by player1234 »

lordnacho wrote: post #64

By and large, I agree with the article. Claude is great and fast at doing low level dev work. Getting the syntax right in some complicated mechanism, executing an edit-execute-readlog loop, making multi file edits. This is exactly why I love it. It's smart enough to do my donkey work. I've revisited the idea that typing speed doesn't matter for programmers. I think it's still an odd thing to judge a candidate on, but…

[dead]

Re: Claude is good at assembling blocks, but still falls apart at creating them

#173

Post by Madmallard »

woeirua wrote: post #133

It's just amazing to me how fast the goal posts are moving. Four years ago, if you had told someone that a LLM would be able to one-shot either of those first two tasks they would've said you're crazy. The tech is moving so fast . I slept on Opus 4.5 because GPT 5 was kind of an air ball, and just started using it in the past few weeks. It's so good. Way better than almost anything that's come before it. It can one-s…

Idk I was using chat gpt 3.5 to do stuff and it was pretty helpful then

Re: Claude is good at assembling blocks, but still falls apart at creating them

#175

Post by 131hn »

groby_b wrote: post #97
disconcision wrote: post #94

I've yet to be convinced by any article, including this one, that attempts to draw boxes around what coding agents are and aren't good at in a way that is robust on a 6 to 12 month horizon. I agree that the examples listed here are relatable, and I've seen similar in my uses of various coding harnesses, including, to some degree, ones driven by opus 4.5. But my general experience with using LLMs for development over…

LLMs are bad at creating abstraction boundaries since inception. People have been calling it out since inception. (Heck, even I got a twitter post somewhere >12 months old calling that out, and I'm not exactly a leading light of the effort) It is in no way size-related. The technology cannot create new concepts/abstractions, and so fails at abstraction. Reliably.

There’s only one way to implement a mission, an algorithm, a task. But there’s an infinity of path, inconsistants, fuzzy and always subjective way to live. Thàt’s our lives, that’s the code LLM are trained on. I do not think, and hope, it will ever change much

Re: Claude is good at assembling blocks, but still falls apart at creating them

#176

Post by Scrapemist »

ChicagoDave wrote: post #114

I have several projects that counter this article. Not sure why, but I’ve extracted clean, readable, well-constructed, and well-tested code. I might write something up at some point, but I can share this: https://github.com/chicagodave/devarch/ New repo with guides for how I use Claude Code.

Interesting. So you put these into the project folder for Claude to follow?

Re: Claude is good at assembling blocks, but still falls apart at creating them

#177

Post by utopiah »

woeirua wrote: post #133

It's just amazing to me how fast the goal posts are moving. Four years ago, if you had told someone that a LLM would be able to one-shot either of those first two tasks they would've said you're crazy. The tech is moving so fast . I slept on Opus 4.5 because GPT 5 was kind of an air ball, and just started using it in the past few weeks. It's so good. Way better than almost anything that's come before it. It can one-s…

> The tech is moving so fast.

Well that's exactly the problem : how can one say that?

The entire process of evaluating what "it" actually does has been a problem from the start. Input text, output text ... OK but what if the training data includes the evaluation? This was ridiculous few years ago but then the scale went from some curated text datasets to... most of the Web as text, to most of the Web as text including transcription from videos, to most of the Web plus some non public databases, to all that PLUS (and that's just cheating) tests that were supposed to be designed to NOT be present elsewhere.

So again, that's the crux of the problem, WHAT does it actually do? Is it "just" search? Is it semantic search with search and replace, is it that plus evaluation that it runs?

Sure the scaffolding becomes bigger, the available dataset becomes larger, the compute available keeps on increasing but it STILL does not answer the fundamental question, namely what is being done. The assumption here is because the output text does solve the question ask, then "it" works, it "solved" the problem. The problem is that by definition the entire setup has been made in order to look as plausible as possible. So it's not luck that it initially appears realistic. It's not luck that it can thus pass some dedicated benchmark, but it is also NOT solving the problem.

So yes sure the "tech" is moving "so fast" but we still can't agree on what it does, we keep on having no good benchmarks, we keep on having that jagged frontier https://www.hbs.edu/faculty/Pages/item.aspx?num=64700 that makes it so challenging to make more meaningful statement than "moving so fast" which sounds like marketing claims.

Re: Claude is good at assembling blocks, but still falls apart at creating them

#178

Post by fastasucan »

bossyTeacher wrote: post #78
lordnacho wrote: post #64

By and large, I agree with the article. Claude is great and fast at doing low level dev work. Getting the syntax right in some complicated mechanism, executing an edit-execute-readlog loop, making multi file edits. This is exactly why I love it. It's smart enough to do my donkey work. I've revisited the idea that typing speed doesn't matter for programmers. I think it's still an odd thing to judge a candidate on, but…

> I feel that I'm learning faster Yes, you are feeling that . But is that real? If I take all LLMs from you right now, is your current you still better than your pre-LLM you? When I dream I feel that I can fly and as long as I am dreaming, this feeling is true. But the subject of this feeling never was.

One guy I work with has little formal training (and mid level experience), but do a lot with LLM's. But in every situation he has to do anything without an LLM he heavily struggles/are not able to anything (say a basic sql query). There is no way someone with his experience and position would still be at that level.

I guess people differ in thinking that is a good or a bad thing. I think it makes up for a huge risk, as he cant really judge good from bad code (or architecture), but his supervisors have put him in a position where he should.

Post reply on HN