Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 4

anthropic.com

Post reply on HN
Print view View original

231–240 of 1001 posts

Re: Claude 4

#231

Post by sebzim4500 »

iLoveOncall wrote: post #63
uludag wrote: post #46

I'm curious what are others priors when reading benchmark scores. Obviously with immense funding at stakes, companies have every incentive to game the benchmarks, and the loss of goodwill from gaming the system doesn't appear to have much consequences. Obviously trying the model for your use cases more and more lets you narrow in on actually utility, but I'm wondering how others interpret reported benchmarks these da…

Hasn't it been proven many times that all those companies cheat on benchmarks? I personally couldn't care less about them, especially when we've seen many times that the public's perception is absolutely not tied to the benchmarks (Llama 4, the recent OpenAI model that flopped, etc.).

I don't think there's any real evidence that any of the major companies are going out of their way to cheat the benchmarks. Problem is that, unless you put a lot of effort into avoiding contamination, you will inevitably end up with details about the benchmark in the training set.

Re: Claude 4

#232

Post by jacob019 »

lxgr wrote: post #206
minimaxir wrote: post #118

An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025) https://docs.anthropic.com/en/docs/about-claude/models/overv...

With web search being available in all major user-facing LLM products now (and I believe in some APIs as well, sometimes unintentionally), I feel like the exact month of cutoff is becoming less and less relevant, at least in my personal experience. The models I'm regularly using are usually smart enough to figure out that they should be pulling in new information for a given topic.

Valid. I suppose the most annoying thing related to the cutoffs, is the model's knowledge of library APIs, especially when there are breaking changes. Even when they have some knowledge of the most recent version, they tend to default to whatever they have seen the most in training, which is typically older code. I suspect the frontier labs have all been working to mitigate this. I'm just super stoked, been waiting for this one to drop.

Re: Claude 4

#233

Post by liorn »

minimaxir wrote: post #118

An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025) https://docs.anthropic.com/en/docs/about-claude/models/overv...

I asked it about Tailwind CSS (since I had problems with Claude not aware of Tailwind 4):

> Which version of tailwind css do you know?

> I have knowledge of Tailwind CSS up to version 3.4, which was the latest stable version as of my knowledge cutoff in January 2025.

Re: Claude 4

#234

Post by drewnick »

diggan wrote: post #59

Anyone with access who could compare the new models with say O1 Pro Mode? Doesn't have to be a very scientific comparison, just some first impressions/thoughts compared to the current SOTA.

I just had some issue with RLS/schema/postgres stuff. Gemini 2.5 Pro swung and missed, and talked a lot with little code, Claude Sonnet 4 solved. O1 Pro Solved. It's definitely random which of these models can solve various problems with the same prompt.

Re: Claude 4

#236

Post by therein »

diggan wrote: post #185
m3kw9 wrote: post #176

Earlier quoted context omitted.

Even that, we don’t know what got updated and what didn’t. Can we assume everything that can be updated is updated?

> Can we assume everything that can be updated is updated? What does that even mean? Of course an LLM doesn't know everything , so it we wouldn't be able to assume everything got updated either. At best, if they shared the datasets they used (which they won't, because most likely it was acquired illegally), you could make some guesses what they tried to update.

> What does that even mean?

I think it is clear what he meant and it is a legitimate question.

If you took a 6 year old and told him about the things that happened in the last year and sent him off to work, did he integrate the last year's knowledge? Did he even believe it or find it true? If that information was conflicting what he knew before, how do we know that the most recent thing he is told he will take as the new information? Will he continue parroting what he knew before this last upload? These are legitimate questions we have about our black box of statistics.

Re: Claude 4

#237

Post by sebzim4500 »

low_tech_punk wrote: post #213

Can anyone help me understand why they changed the model naming convention? BEFORE: claude-3-7-sonnet AFTER: claude-sonnet-4

Because we are going to get AGI before an AI company can consistently name their models.

Re: Claude 4

#238

Post by j_maffe »

comova wrote: post #98
energy123 wrote: post #79

> Finally, we've introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. This summarization is only needed about 5% of the time—most thought processes are short enough to display in full. This is not better for the user. No users want this. If you're doing this to prevent competitors training on your thought traces then fine. But if you really believe this is…

I believe this is to improve performance by shortening the context window for long thinking processes. I don't think this is referring to real-time summarizing for the users' sake.

> I don't think this is referring to real-time summarizing for the users' sake.

That's exactly what it's referring to.

Re: Claude 4

#239

Post by mikepurvis »

9dev wrote: post #207
yosito wrote: post #169

Earlier quoted context omitted.

Personally, I don't believe AI is ever going to get to that level. I'd love to be proven wrong, but I really don't believe that an LLM is the right tool for a job that requires novel thinking about out of the ordinary problems like all the weird edge cases and poor documentation that comes up when trying to upgrade old software.

Actually, I think the opposite: Upgrading a project that needs dependency updates to new major versions—let’s say Zod 4, or Tailwind 3—requires reading the upgrade guides and documentation, and transferring that into the project. In other words, transforming text. It’s thankless, stupid toil. I’m very confident I will not be doing this much more often in my career.

Absolutely, this should be exactly the kind of task a bot should be perfect for. There's no abstraction, no design work, no refactoring, no consideration of stakeholders, just finding instances of whatever is old and busted and changing it for the new hotness.

Re: Claude 4

#240

Post by danielbln »

iLoveOncall wrote: post #54

I can't think of more boring than marginal improvements on coding tasks to be honest. I want GenAI to become better at tasks that I don't want to do, to reduce the unwanted noise from my life. This is when I'll pay for it, not when they found a new way to cheat a bit more the benchmarks. At work I own the development of a tool that is using GenAI, so of course a new better model will be beneficial, especially because…

What if coding is that unwanted task? Also, what are the tasks you are referring to, specifically?
Post reply on HN