221–230 of 1001 posts
An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025) https://docs.anthropic.com/en/docs/about-claude/models/overv...
Both Sonnet and Opus 4 say Joe Biden is president and claim their knowledge cutoff is "April 2024".
Claude 3.8 wrote me some code this morning, and I was running into a bug. I switched to 4 and gave it its own code. It pointed out the bug right away and fixed it. So an upgrade for me :-)
Did you try 3.7 for debugging first? Just telling it there's a bug can be enough.
I wish someone focused on making the models give better answers about the Beatles or Herodotus...
Is this really worthy of a claude 4 label? Was there a new pre-training run? Cause this feels like 3.8... only swe went up significantly, and that as we all understand by now is done by cramming on specific post training data and doesn't generalize to intelligence. The agentic tooluse didn't improve and this says to me that it's not really smarter.
Wonder why they renamed it from Claude (e.g. Claude 3.7 Sonnet) to Claude (Claude Opus 4).
Earlier quoted context omitted.
You might be able to ask it what it knows.
So something's odd there. I asked it "Who won Super Bowl LIX and what was the winning score?" which was in February and the model replied "I don't have information about Super Bowl LIX (59) because it hasn't been played yet. Super Bowl LIX is scheduled to take place in February 2025.".
I imagine there's a lot more data pointing to the super bowl being upcoming, then the super bowl concluding with the score.
Gonna be scary when bot farms are paid to make massive amounts of politically motivated false content (specifically) targeting future LLMs training
> Finally, we've introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. This summarization is only needed about 5% of the time—most thought processes are short enough to display in full. This is not better for the user. No users want this. If you're doing this to prevent competitors training on your thought traces then fine. But if you really believe this is…
I believe this is to improve performance by shortening the context window for long thinking processes. I don't think this is referring to real-time summarizing for the users' sake.
Earlier quoted context omitted.
Claude Plays Pokemon was the original concept and inspiration behind "Gemini Plays Pokemon". Gemini arguably only did better because it had access to a much better agent harness and was being actively developed during the run. See: https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-...
Not sure "original concept" is quite right, given it had been tried earlier, e.g. here's a 2023 attempt to get gpt-4-vision to play pokemon, (it didn't really work, but it's clearly "the concept") https://x.com/sidradcliffe/status/1722355983643525427
[1] https://community.aws/content/2gbBSofaMK7IDUev2wcUbqQXTK6/ca...
Earlier quoted context omitted.
So I decided to try Claude 4 Sonnet against my "Given a list of 1 million random integers between 1 and 100,000, find the difference between the smallest and the largest numbers whose digits sum up to 30." benchmark I tested against Claude 3.5 Sonnet: https://news.ycombinator.com/item?id=42584400 The results are here ( https://gist.github.com/minimaxir/1bad26f0f000562b1418754d67... ) and it utterly crushed the proble…
> although it's possible Claude 4 was trained on that discussion lol Almost guaranteed, especially since HN tends to be popular in tech circles, and also trivial to scrape the entire thing in a couple of hours via the Algolia API. Recommendation for the future: keep your benchmarks/evaluations private, as otherwise they're basically useless as more models get published that are trained on your data. This is what I do…
> Almost guaranteed, especially since HN tends to be popular in tech circles, and also trivial to scrape the entire thing via the Algolia API.
I am wondering if this could be cleverly exploited.
Nice to see that Sonnet performs worse than o3 on AIME but better on SWE-Bench. Often, it's easy to optimize math capabilities with RL but much harder to crack software engineering. Good to see what Anthropic is focusing on.