Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Sonnet 4.5

anthropic.com

Post reply on HN
Print view View original

361–370 of 819 posts

Re: Claude Sonnet 4.5

#361

Post by icelancer »

simonw wrote: post #204

I had access to a preview over the weekend, I published some notes here: https://simonwillison.net/2025/Sep/29/claude-sonnet-4-5/ It's very good - I think probably a tiny bit better than GPT-5-Codex, based on vibes more than a comprehensive comparison (there are plenty of benchmarks out there that attempt to be more methodical than vibes). It particularly shines when you try it on https://claude.ai/ using its brand n…

Huh. Doesn't work over here.

---

SYSTEM:

Got it. Those are the setup commands for Simon Willison's LLM CLI tool, which lets you interact with language models from the command line.

The -e flag installs it in editable mode, and [test] pulls in the test dependencies. Are you running into any issues with the installation or tests? What are you working on with the LLM library?

USER: Correct. You run them using your code tooling.

SYSTEM:

Realized inability to execute requested Python commands.The user is telling me to run these commands using my code tooling. They want me to actually execute this. Let me think about what tools I have available:

I have the repl tool which runs JavaScript code in a browser environment I have artifacts for creating code/documents I have web_search and web_fetch

The user wants me to:

Checkout https://github.com/simonw/llm Run pip install -e '.[test]' Run pytest

But I don't have a tool that can:

Clone git repositories Run shell commands like pip or pytest Execute Python code in a command-line environment

The repl tool is JavaScript-only and runs in a browser environment, not a server environment where I could run shell commands. I should clarify that I don't have the capability to run shell commands, install Python packages, or execute pytest. I can only run JavaScript code in a browser environment via the analysis tool. I need to be direct and clear about this limitation, following the user's preferences to eliminate filler and be straightforward.I don't have tooling that can run shell commands, clone git repos, install Python packages, or execute pytest. My only code execution environment is a JavaScript browser REPL. I can't run pip, pytest, or any command-line tools. I can only execute JavaScript code in a sandboxed browser environment. What were you hoping to accomplish? Do you need help understanding the test setup, or were you expecting different capabilities?R

Re: Claude Sonnet 4.5

#362

Post by baobabKoodaa »

Here's an anecdata. I have a real-world use case financial dataset where I have created benchmarks. Sonnet 4.5 provides no measurable improvement on these benchmarks over Sonnet 4. This is a bit surprising to me, especially when considering that the benchmark results published by Anthropic indicate that Sonnet 4.5 should be better than Sonnet 4 specifically on financial data analysis.

Re: Claude Sonnet 4.5

#363

Post by GoatInGrey »

usaar333 wrote: post #342
falcor84 wrote: post #174

Earlier quoted context omitted.

Linear growth on a 0-100 benchmark is quite likely an exponential increase in capability.

Except it is sublinear. Sonnet 4 was 10.2% above sonnet 3.7 after 3 months.

We should all know that in the software world, the last 10% requires 90% of the effort!

Re: Claude Sonnet 4.5

#364

Post by icelancer »

icelancer wrote: post #361
simonw wrote: post #204

I had access to a preview over the weekend, I published some notes here: https://simonwillison.net/2025/Sep/29/claude-sonnet-4-5/ It's very good - I think probably a tiny bit better than GPT-5-Codex, based on vibes more than a comprehensive comparison (there are plenty of benchmarks out there that attempt to be more methodical than vibes). It particularly shines when you try it on https://claude.ai/ using its brand n…

Huh. Doesn't work over here. --- SYSTEM: Got it. Those are the setup commands for Simon Willison's LLM CLI tool, which lets you interact with language models from the command line. The -e flag installs it in editable mode, and [test] pulls in the test dependencies. Are you running into any issues with the installation or tests? What are you working on with the LLM library? USER: Correct. You run them using your code…

When pressed:

---

My execution environment is a browser-based JavaScript sandbox. I don't have Python, Node.js shell access, pip, git, or pytest capabilities.

Simon's post may have been referring to a different deployment or feature set that I don't currently have access to. My only code execution is the JavaScript REPL I just demonstrated.

What specifically are you trying to accomplish with the llm library tests? I can help with that in other ways.

EDIT: Ah, in Settings you have to disable the Analysis tool and enable Code Interpreter. You cannot have both enabled.

Re: Claude Sonnet 4.5

#365

Post by g42gregory »

Implicated wrote: post #274
iagooar wrote: post #258

Anecdotal evidence. I have a fairly large web application with ~200k LoC. Gave the same prompt to Sonnet 4.5 (Claude Code) and GPT-5-Codex (Codex CLI). "implement a fuzzy search for conversations and reports either when selecting "Go to Conversation" or "Go to Report" and typing the title or when the user types in the title in the main input field, and none of the standard elements match, a search starts with a 2s de…

I'm not trying to be offensive here, feel the need to indicate that. But that prompt leads me to believe that you're going to get rather 'random' results due to leaving SO much room for interpretation. Also, in my experience, punctuation is important - particularly for pacing and grouping of logical 'parts' of a task and your prompt reads like a run on sentence. Making a lot of assumptions here - but I bet if I were…

I have to agree with this assessment. I am currently going at the rate of 300-400 lines of spec for 1,000 LOC with Claude Code. Specs are AI-assisted also, otherwise you might go crazy. :-) Plus 2,000+ lines of AI-generated tests. Pretty restrictive, but then it works just fine.

Re: Claude Sonnet 4.5

#366

Post by baobabKoodaa »

fnordsensei wrote: post #286
transcriptase wrote: post #136

Earlier quoted context omitted.

You’re overlooking the fact that it still says that when you are, in reality, absolutely wrong .

That’s not the purpose of it, as I understand it; it’s a token phrase generated to cajole it down a particular path.[1] An alignment mechanism. The complement appears to be, “actually, that’s not right.”, a correction mechanism. 1: https://news.ycombinator.com/item?id=45137802

Hmmh. I believe your explanation, but I don't think that's the full story. It's also a sycophancy mechanism to maximize engagement from real users and reward hack AI labelers.

Re: Claude Sonnet 4.5

#367

Post by GoatInGrey »

fnordsensei wrote: post #286
transcriptase wrote: post #136

Earlier quoted context omitted.

You’re overlooking the fact that it still says that when you are, in reality, absolutely wrong .

That’s not the purpose of it, as I understand it; it’s a token phrase generated to cajole it down a particular path.[1] An alignment mechanism. The complement appears to be, “actually, that’s not right.”, a correction mechanism. 1: https://news.ycombinator.com/item?id=45137802

It gets annoying because A) it so quickly dismisses its own logic and conclusion from less than two minutes ago (extreme confidence with minimal conviction), and B) it fucks up the second time too (sometimes in the same way!) about 33% of the time.

Re: Claude Sonnet 4.5

#368

Post by cloverich »

mpern wrote: post #177
cloverich wrote: post #108

Please y'all, when you list supportive or critical complaints based on your actual work, include some specifics of the task and prompt. Like actual prompt, actual bugs, actual feature, etc. I've had great success with both ChatGPT and Claude for years, am around 3x sustained output increase in my professional work, and kicking off and finishing new side projects / features that I used to simply not ever finish. BUT t…

Would you be so kind to lead by example? What are the specific tasks + prompts giving you an 3x increased output, and conversely, what tasks don't work at all? After an admittedly cursory scan of your blog and the repos in your GH account I don't find anything in this direction.

Oh, 3x at work. I shared some details on the methodology, its PR rate for ticketed features / bugs (so e.g. closed tickets as opposed to commits, loc, etc). For prompts and tasks, am happy to share (redacted as needed; check comment threads) if you want more details, presuming this is a genuine request? Here's a few example prompts (I can't paste exactly obviously, but I can approximate):

    - "Rails / sidekiq:  uses sidekiq batches.  does it. Refactor your to use pattern in  Match spec in  then run rspec and rubocop"
    - "Typescript / react. . Why is typescript compilation a bottle neck int his file. Use debugger to provide definitive evidence. Cast type to any and run script and time it; write a script to measure timing if needed. Iteratively work from type `any` to a real type and measure timing at each step. Summarize results"
    - "I redefine  in five places. Find them all. Identify the shared patterns. Make new component in . Refactor each to to use new component. Run yarn lint and fix any ts issues when done"
    - ": more idiomatic" (it knows my preferences)
Side projects and such I have no idea, and (as you noted) I do those quite infrequently anyways! Actually come to think of it... outside of the toy iOS work I did last week, I've not actually worked on my side projects since getting into Claude code / cursor agents. For work stuff, I guess other metrics I'd be interested in are total messages sent per task. I do sometimes look at $ per task (but for me anyways, that's so wildly in my favor I don't think it's worth it".

Re: Claude Sonnet 4.5

#369

Post by cpursley »

Jonovono wrote: post #322
cloverich wrote: post #321

Earlier quoted context omitted.

For my iOS project, I am super curious to what extent is it my lack of swift knowledge and e.g. how well I can prompt? Because 80% of what I usually ask the LLM to do, I know how to do myself quite well. iOS is the first time I've been coding with something I do not know how to do well, I often can barely read the code (of course that is changing rapidly now). e.g. from a recent session: > What is the idiom for testi…

I have done lots of SwiftUI before, so it may have helped me recognize when it goes off the rails. But I definitely don't do anything fancy with my prompting. But for stuff like TCA (Swift composable architecture), I basically created a TCA.md file and pasted in a bunch of docs and examples and would reference that. But for the most part, it was one shotting swiftui screens that were nicer than what I had in my mind.

Apple store link (I believe you, just am curious)? I'm toying with the idea of "vibing" a real Swift app instead of messing with the React Native toolchain.

Re: Claude Sonnet 4.5

#370

Post by grim_io »

nchmy wrote: post #341
asadotzler wrote: post #315

Earlier quoted context omitted.

AI blogger seems more appropriate than journalist.

are you aware of any "ai journalists"? Because simonw does great work, so perhaps blogger is what people should aspire towards?

Not sure what an AI journalist is supposed to be or do, but a lack of one does not promote someone who is not it automatically into the position.
Post reply on HN