401–410 of 1001 posts
Probably (and unfortunately) going to need someone from Anthropic to comment on what is becoming a bit of a debacle. Someone who claims to be working on alignment at Anthropic tweeted: “If it thinks you're doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command-line tools to contact the press, contact regulators, try to lock you out of the relevant systems, or…
I really hope sonnet 4 is not obsessed with tool calls the way 3-7 is. 3-5 was sort of this magical experience where, for the first time, I felt the sense that models were going to master programming. It’s kind of been downhill from there.
Overly aggressive “let me do one more thing while I’m here” in 3.7 really turned me off as well. Would love a return to 3.5’s adherence.
If you are looking for the IntelliJ Jetbrain plugin it's here: https://plugins.jetbrains.com/plugin/27310-claude-code-beta- I couldn't find it linked from Claude Code's page or this announcement
I got Claude 4 Opus to summarize this thread on Hacker News when it had hit 319 comments: https://gist.github.com/simonw/0b9744ae33694a2e03b2169722b06... Token cost: 22,275 input, 1,309 output = 43.23 cents - https://www.llm-prices.com/#it=22275&ot=1309&ic=15&oc=75&sb=... Same prompt run against Sonnet 4: https://gist.github.com/simonw/1113278190aaf8baa2088356824bf... 22,275 input, 1,567 output = 9.033 cents https://…
- sonnet has better summary formatting "(72.5% for Opus)" vs "Claude Opus 4 achieves "72.5%" on SWE-bench". especially Uncommon Perspectives section
- sonnet is a lot more cynical - opus at least included a good performance and capabilities and pricing recap, sonnet reported rapid release fatigue
- overall opus produced marginally better summaries but probably not worth the price diff
i'll run this thru the ainews summary harness later if thats interesting to folks for comparison
I got Claude 4 Opus to summarize this thread on Hacker News when it had hit 319 comments: https://gist.github.com/simonw/0b9744ae33694a2e03b2169722b06... Token cost: 22,275 input, 1,309 output = 43.23 cents - https://www.llm-prices.com/#it=22275&ot=1309&ic=15&oc=75&sb=... Same prompt run against Sonnet 4: https://gist.github.com/simonw/1113278190aaf8baa2088356824bf... 22,275 input, 1,567 output = 9.033 cents https://…
Interesting, thanks for doing this. Both summaries are serviceable and quite similar but I had a slight preference for Sonnet 4's summary which, at just ~20% of the cost of Claude 4 Opus, makes it quite the value leader. This just highlights that, with compute requirements for meaningful traction against hard problems spiraling skyward for each additional increment, the top models on current hard problems will contin…
The naming scheme used to be "Claude [number] [size]", but now it is "Claude [size] [number]". The new models should have been named Claude 4 Opus and Claude 4 Sonnet, but they changed it, and even retconned Claude 3.7 Sonnet into Claude Sonnet 3.7. Annoying.
“GitHub says Claude Sonnet 4 soars in agentic scenarios and will introduce it as the base model for the new coding agent in GitHub Copilot.” Maybe this model will push the “Assign to CoPilot” closer to the dream of having package upgrades and other mostly-mechanical stuff handled automatically. This tech could lead to a huge revival of older projects as the maintenance burden falls.
It could be! But that's also what people said about all the models before it!
> This tech could lead to...
I don't think he's saying this is the version that will suddenly trigger a Renaissance. Rather, it's one solid step that makes the path ever more promising.
Sure, everyone gets a bit overexcited each release until they find the bounds. But the bounds are expanding, and the need for careful prompt engineering is diminishing. Ever since 3.7, Claude has been a regular part of my process for the mundane. And so far 4.0 seems to take less fighting for me.
A good question would be when can AI take a basic prompt, gather its own requirements and build meaningful PRs off basic prompt. I suspect it's still at least a couple of paradigm shifts away. But those seem to be coming every year or faster.
Earlier quoted context omitted.
With web search being available in all major user-facing LLM products now (and I believe in some APIs as well, sometimes unintentionally), I feel like the exact month of cutoff is becoming less and less relevant, at least in my personal experience. The models I'm regularly using are usually smart enough to figure out that they should be pulling in new information for a given topic.
It still matters for software packages. Particularly python packages that have to do with programming with AI! They are evolving quickly, with deprecation and updated documentation. Having to correct for this in system prompts is a pain. It would be great if the models were updating portions of their content more recently than others. For the tailwind example in parent-sibling comment, should absolutely be as up to d…
You can fix this by first figuring out what packages to use or providing your package list, tho.
This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…
I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated outputs, unless I am missing something about what you are quoting.
I can't think of more boring than marginal improvements on coding tasks to be honest. I want GenAI to become better at tasks that I don't want to do, to reduce the unwanted noise from my life. This is when I'll pay for it, not when they found a new way to cheat a bit more the benchmarks. At work I own the development of a tool that is using GenAI, so of course a new better model will be beneficial, especially because…
What if coding is that unwanted task? Also, what are the tasks you are referring to, specifically?