Claude 4

Post by toomuchtodo » Thu, May 22, 2025, 7:02 PM UTC

bredren wrote: post #270

lxgr wrote: post #206

Earlier quoted context omitted.

With web search being available in all major user-facing LLM products now (and I believe in some APIs as well, sometimes unintentionally), I feel like the exact month of cutoff is becoming less and less relevant, at least in my personal experience. The models I'm regularly using are usually smart enough to figure out that they should be pulling in new information for a given topic.

It still matters for software packages. Particularly python packages that have to do with programming with AI! They are evolving quickly, with deprecation and updated documentation. Having to correct for this in system prompts is a pain. It would be great if the models were updating portions of their content more recently than others. For the tailwind example in parent-sibling comment, should absolutely be as up to d…

Post by losvedir » Thu, May 22, 2025, 7:02 PM UTC

minimaxir wrote: post #179

KaoruAoiShiho wrote: post #61

Is this really worthy of a claude 4 label? Was there a new pre-training run? Cause this feels like 3.8... only swe went up significantly, and that as we all understand by now is done by cramming on specific post training data and doesn't generalize to intelligence. The agentic tooluse didn't improve and this says to me that it's not really smarter.

So I decided to try Claude 4 Sonnet against my "Given a list of 1 million random integers between 1 and 100,000, find the difference between the smallest and the largest numbers whose digits sum up to 30." benchmark I tested against Claude 3.5 Sonnet: https://news.ycombinator.com/item?id=42584400 The results are here ( https://gist.github.com/minimaxir/1bad26f0f000562b1418754d67... ) and it utterly crushed the proble…

Post by sandspar » Thu, May 22, 2025, 7:04 PM UTC

douglasisshiny wrote: post #175

candiddevmike wrote: post #68

Earlier quoted context omitted.

People's interpretation of benchmarks will largely depend on whether they believe they will be better or worse off by GenAI taking over SWE jobs. Think you'd need someone outside the industry to weigh in to have a real, unbiased view.

Or someone who has been a developer for a decade plus trying to use these models on actual existing code bases, solving specific problems. In my experience, they waste time and money.

Post by guywithahat » Thu, May 22, 2025, 7:04 PM UTC

lxgr wrote: post #206

minimaxir wrote: post #118

An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025) https://docs.anthropic.com/en/docs/about-claude/models/overv...

With web search being available in all major user-facing LLM products now (and I believe in some APIs as well, sometimes unintentionally), I feel like the exact month of cutoff is becoming less and less relevant, at least in my personal experience. The models I'm regularly using are usually smart enough to figure out that they should be pulling in new information for a given topic.

Post by ketzo » Thu, May 22, 2025, 7:05 PM UTC

deciduously wrote: post #147

waleedlatif1 wrote: post #108

I really hope sonnet 4 is not obsessed with tool calls the way 3-7 is. 3-5 was sort of this magical experience where, for the first time, I felt the sense that models were going to master programming. It’s kind of been downhill from there.

This feels like more of a system prompt issue than a model issue?