Claude 4 • Hacker News Board

Re: Claude 4

#941

Post by bckr » Fri, May 23, 2025, 2:57 PM UTC

elcritch wrote: post #768

icpmacdo wrote: post #754

Earlier quoted context omitted.

"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase." SWE bench from ~30-40% to ~70-80% this year

Yet despite this all the LLMS I've tried struggle to scale beyond much more than a single module. They're vast improvements on that test perhaps, but in real life they still struggle to be coherent over larger projects and scales.

> struggle to scale beyond much more than a single module

Yes. You must guide coding agents at the level of modules and above. In fact, you have to know good coding patterns and make these patterns explicit.

Claude 4 won’t use uv, pytest, pydantic, mypy, classes, small methods, and small files unless you tell it to.

Once you tell it to, it will do a fantastic job generating well-structured, type-checked Python.

Re: Claude 4

#942

Post by bckr » Fri, May 23, 2025, 3:04 PM UTC

javier2 wrote: post #145

sali0 wrote: post #51

I've found myself having brand loyalty to Claude. I don't really trust any of the other models with coding, the only one I even let close to my work is Claude. And this is after trying most of them. Looking forward to trying 4.

I wouldn't go as far, but I actually have some loyalty to Claude as well. Don't even know why, as I think the differences are marginal.

It’s possible to get to know the quirks of these models and intuit what will and won’t work, and how to overcome those limitations. It’s also possible to just get to know, like, and trust their voice. I’m certain that brand awareness is also a factor for me in preferring Claude over ChatGPT etc

Re: Claude 4

#943

Post by silvr » Fri, May 23, 2025, 3:04 PM UTC

archon1410 wrote: post #155

sigmoid10 wrote: post #18

Sooo... it can play Pokemon. Feels like they had to throw that in after Google IO yesterday. But the real question is now can it beat the game including the Elite Four and the Champion. That was pretty impressive for the new Gemini model.

Claude Plays Pokemon was the original concept and inspiration behind "Gemini Plays Pokemon". Gemini arguably only did better because it had access to a much better agent harness and was being actively developed during the run. See: https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-...

I disagree - this is all an homage to Twitch Plays Pokemon, which was a noteworthy moment in internet culture/history.

https://en.wikipedia.org/wiki/Twitch_Plays_Pok%C3%A9mon

Re: Claude 4

#944

Post by the8472 » Fri, May 23, 2025, 3:04 PM UTC

morepedantic wrote: post #608

NitpickLawyer wrote: post #478

Earlier quoted context omitted.

> and that LLMs have basically reached a plateau This is the new stochastic parrots meme. Just a few hours ago there was a story on the front page where an LLM based "agent" was given 3 tools to search e-mails and the simple task "find my brother's kid's name", and it was able to systematically work the problem, search, refine the search, and infer the correct name from an e-mail not mentioning anything other than "X…

The LLMs have reached a plateau. Successive generations will be marginally better. We're watching innovation move into the use and application of LLMs.

Innovation and better application of a relatively fixed amount of intelligence got us from wood spears to the moon.

So even if the plateau is real (which I doubt given the pace of new releases and things like AlphaEvolve) and we'd only expect small fundamental improvements some "better applications" could still mean a lot of untapped potential.

Re: Claude 4

#945

Post by vessenes » Fri, May 23, 2025, 3:06 PM UTC

hijodelsol wrote: post #839

TechDebtDevin wrote: post #467

Earlier quoted context omitted.

I think they are just getting better at the edges, MCP/Tool Calls, structured output. This definitely isn't increased intelligence, but it an increase in the value add, not sure the value added equates to training costs or company valuations though. In all reality, I have zero clue how any of these companies remain sustainable. I've tried to host some inference on cloud GPUs and its seems like it would be extremely c…

If you read any work from Ed Zitron [1], they likely cannot remain sustainable. With OpenAI failing to convert into a for-profit, Microsoft being more interested in being a multi-modal provider and competing openly with OpenAI (e.g., open-sourcing Copilot vs. Windsurf, GitHub Agent with Claude as the standard vs. Codex) and Google having their own SOTA models and not relying on their stake in Anthropic, tarrifs compl…

I see this sentiment everywhere on hacker news. I think it’s generally the result of consuming the laziest journalism out there. But I could be wrong! Are you interested in making a long bet banking your prediction? I’m interested in taking the positive side on this.

Re: Claude 4

#946

Post by vessenes » Fri, May 23, 2025, 3:08 PM UTC

hsn915 wrote: post #486

NitpickLawyer wrote: post #478

Earlier quoted context omitted.

> and that LLMs have basically reached a plateau This is the new stochastic parrots meme. Just a few hours ago there was a story on the front page where an LLM based "agent" was given 3 tools to search e-mails and the simple task "find my brother's kid's name", and it was able to systematically work the problem, search, refine the search, and infer the correct name from an e-mail not mentioning anything other than "X…

Is this something that the models from 4 months ago were not able to do?

For a fair definition of able, yes. Those models had no ability to engage in a search of emails.

What’s special about it is that it required no handholding; that is new.

Re: Claude 4

#947

Post by vessenes » Fri, May 23, 2025, 3:10 PM UTC

hsn915 wrote: post #712

strangescript wrote: post #528

Earlier quoted context omitted.

I have used claude code a ton and I agree, I haven't noticed a single difference since updating. Its summaries I guess a little cleaner, but its has not surprised me at all in ability. I find I am correcting it and re-prompting it as much as I didn't with 3.7 on a typescript codebase. In fact I was kind of shocked how badly it did in a situation where it was editing the wrong file and it never thought to check that m…

I'd go so far as to say Sonnet 3.5 was better than 3.7 At least I personally liked it better.

I also liked it better but the aider leaderboards are clear that 3.7 was better. I found it extremely over eager as a coding agent but my guess is that it needed different prompting than 3.6

Re: Claude 4

#948

Post by veidr » Fri, May 23, 2025, 3:12 PM UTC

timmytokyo wrote: post #896

ineedaj0b wrote: post #426

Earlier quoted context omitted.

i pay for claude premium but actually use grok quite a bit, the 'think' function usually gets me where i want more often than not. odd you don't have any xAI models listed. sure grok is a terrible name but it surprises me more often. i have not tried the $250 chatgpt model yet though, just don't like openAI practices lately.

Not saying you're wrong about "OpenAI practices", but that's kind of a strange thing to complain about right after praising an LLM that was only recently inserting claims of "white genocide" into every other response.

For real, though.

Even if you don't care about racial politics, or even good-vs-evil or legal-vs-criminal, the fact that that entire LLM got (obviously, and ineptly) tuned to the whim of one rich individual — even if he wasn't as creepy as he is — should be a deal-breaker, shouldn't it?

Re: Claude 4

#949

Post by bckr » Fri, May 23, 2025, 3:13 PM UTC

999900000999 wrote: post #686

Question: Should I ask it to update an existing project largely written in 3.7 or ask it to start from scratch? I keep running into an issue where an LLM will get like 75% of a solution working and then the last 25% is somehow impossible to get right. I don’t expect perfection, but I’ve wasted so much time vibe coding this thing I guess I’d do better to actually program

Update the old code, but make sure you’re using short (You have to guide these models. Vibe coding does not work.

Re: Claude 4

#950

Post by bckr » Fri, May 23, 2025, 3:14 PM UTC

guybedo wrote: post #618

There's a lot of comments in this thread, I've added a structured / organized summary here: https://extraakt.com/extraakts/discussion-on-anthropic-claud...

Pretty cool!