Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Sonnet 4.5

anthropic.com

Post reply on HN
Print view View original

771–780 of 819 posts

Re: Claude Sonnet 4.5

#771

Post by user34283 »

doix wrote: post #711
techpression wrote: post #694

It took me one question to have it spit out a completely dreamt up codebase, complete with emojis, promises of solutions and fixing all my problems, and of course nothing of it worked. It was a very simple question about something very well documented (Oban timeouts). I doubt LLM benchmarks more and more, what are they even testing?

> It was a very simple question about something very well documented (Oban timeouts). It's some 3rd party thing for Elixir, a niche within a niche. I wouldn't expect an LLM to do well there. > I doubt LLM benchmarks more and more, what are they even testing? Probably testing by asking it to solve a problem with python or (java|type)script. Perhaps not even specifying a language and watching it generate a generic Reac…

Expectations vary wildly.

Sometimes people expect to use LLMs to unearth hard to find information.

In reality, LLMs seem to quickly fall apart when you go from ubiquitous libraries with 200k stars on GitHub to one with "just" 1k stars.

What makes the situation worse is the way LLMs fail. Hallucinations where it goes "my usage example did not work because you are on the wrong version of the library/using the wrong SDK" etc. are super common in this scenario. This leads to further time wasted trying to apply reasonably plausible fixes that are entirely hallucinated.

Re: Claude Sonnet 4.5

#772

Post by whywhywhywhy »

simonw wrote: post #743
whywhywhywhy wrote: post #715

Earlier quoted context omitted.

Kinda pointless listening to the opinions of people who've used previews because it's not gonna be the same model you'll experience once it gets downgraded to be viable under mass use and the benchmarks influencers use are all in the training data now and tested internally so any sort of testing like pelicans on bikes is just PR at this point.

I learned that lesson from GPT-5, where the preview was weeks long and the models kept changing during that period. This Claude preview lasted from Friday to Monday so I was less worried about major model changes. I made sure to run the pelican benchmark against the model after 10am on Monday (the official release date) just to be safe. The only thing I published that I ran against the preview model was the Claude co…

Yesterday someone posted an example of the same prompt but changing it to a human and it was basically trash, the example you've posted actually looks good all things considered. So yeah I do think its something they train on, same way they train on things in the benchmarks.

Re: Claude Sonnet 4.5

#773

Post by unshavedyak »

alecco wrote: post #381
iagooar wrote: post #258

Anecdotal evidence. I have a fairly large web application with ~200k LoC. Gave the same prompt to Sonnet 4.5 (Claude Code) and GPT-5-Codex (Codex CLI). "implement a fuzzy search for conversations and reports either when selecting "Go to Conversation" or "Go to Report" and typing the title or when the user types in the title in the main input field, and none of the standard elements match, a search starts with a 2s de…

Try adding ultrathink to the prompt, hit enter, and play this song in the background: [1] https://www.reddit.com/r/ClaudeAI/comments/1mgwohq/ultrathin...

Sidenote, Claude Code now has a thinking toggle. Maybe it did already, but that's new to me. I see better results by just leaving that toggle on.

Re: Claude Sonnet 4.5

#774

Post by hu3 »

StarterPro wrote: post #689

Once the bottom falls out of ai, will programming be seen as a marketable skill again?

I think so. Because systems require a lot of knowledge to create and maintain without breaking.

How many years till AI can be trusted to deploy changes to production without supervision?

Maybe never.

Re: Claude Sonnet 4.5

#775

Post by zamadatix »

MichealCodes wrote: post #580
zamadatix wrote: post #560

Earlier quoted context omitted.

> The benchmarks are not typically ongoing, we do not often see comparisons between week 1 and week 8 You don't need to compare "A (Week 1)" to "A (Week 8)" to be able to show "B (Week 1)" is genuinely x% better than "A (Week 1)".

As I said sprinkle a bit of benchmarks polluting the training and you have your loop. Each iteration will be better at benchmarks if that's the goal and that goal/context reinforces.

Sprinkling in benchmark training isn't a loop, it's just plain cheating. Regardless, not all of these benchmarks are public and, even with mass collusion across the board, it wouldn't make sense only open weight LLMS have been improving.

Re: Claude Sonnet 4.5

#776

Post by 0xbadcafebee »

ranguna wrote: post #687
0xbadcafebee wrote: post #457

Claude doesn't know how to calculate realistic minimum voltages for solar arrays w/MPPT chargers. ChatGPT does. Prompt: "Can I use two strings of four Phono Solar PS440M8GFH solar panels with a EG4 12kPV Hybrid Inverter? I want to make sure that there will not be an issue any time of year. New York upstate." Claude 4.5: Returns within a few seconds. Does not find the PV panel specs, so it asks me if I want it to sear…

This HN post is about claude 4.5 and you come here speaking about how "claude" does not give you satisfactory answer when, most likely, you didn't even try claude 4.5 in the first place. Claude 4.5 after a few web searches and running a couple python scripts for analysis: Yes, your configuration should work! Based on my analysis, two strings of four Phono Solar PS440M8GFH panels will be compatible with the EG4 12kPV…

I said "Claude 4.5" in my comment

Re: Claude Sonnet 4.5

#777

Post by simonw »

whywhywhywhy wrote: post #772
simonw wrote: post #743

Earlier quoted context omitted.

I learned that lesson from GPT-5, where the preview was weeks long and the models kept changing during that period. This Claude preview lasted from Friday to Monday so I was less worried about major model changes. I made sure to run the pelican benchmark against the model after 10am on Monday (the official release date) just to be safe. The only thing I published that I ran against the preview model was the Claude co…

Yesterday someone posted an example of the same prompt but changing it to a human and it was basically trash, the example you've posted actually looks good all things considered. So yeah I do think its something they train on, same way they train on things in the benchmarks.

The easy way to tell is to try it yourself - run "Generate an SVG of a pelican riding a bicycle" and then try "Generate an SVG of an otter riding a skateboard" and see if the quality of the images seems similar.

Re: Claude Sonnet 4.5

#778

Post by bobbylarrybobby »

iagooar wrote: post #258

Anecdotal evidence. I have a fairly large web application with ~200k LoC. Gave the same prompt to Sonnet 4.5 (Claude Code) and GPT-5-Codex (Codex CLI). "implement a fuzzy search for conversations and reports either when selecting "Go to Conversation" or "Go to Report" and typing the title or when the user types in the title in the main input field, and none of the standard elements match, a search starts with a 2s de…

It's also my experience that Claude loves to reimplement the wheel instead of reading code to look for an existing implementation of what it wants to do.

Re: Claude Sonnet 4.5

#779

Post by simonw »

tripzilch wrote: post #758
simonw wrote: post #588

Earlier quoted context omitted.

If they do that I'll go back to writing about them after they ship. Not a big loss for me at all.

I get it, you would trust yourself if you said that, but it doesn't really matter whether you say that or not, what counts for your ongoing credibility if you will preface every future blog post with, whether you got special access, a special deal, sponsorship, or the fact that you didn't get any of those things. You're a reviewer. This is how reviewers stay credible. If you don't disclose your relationship with the…

I recently started doing that: https://simonwillison.net/about/#disclosures and https://simonwillison.net/tags/disclosures/

Re: Claude Sonnet 4.5

#780

Post by simonw »

user34283 wrote: post #771
doix wrote: post #711

Earlier quoted context omitted.

> It was a very simple question about something very well documented (Oban timeouts). It's some 3rd party thing for Elixir, a niche within a niche. I wouldn't expect an LLM to do well there. > I doubt LLM benchmarks more and more, what are they even testing? Probably testing by asking it to solve a problem with python or (java|type)script. Perhaps not even specifying a language and watching it generate a generic Reac…

Expectations vary wildly. Sometimes people expect to use LLMs to unearth hard to find information. In reality, LLMs seem to quickly fall apart when you go from ubiquitous libraries with 200k stars on GitHub to one with "just" 1k stars. What makes the situation worse is the way LLMs fail. Hallucinations where it goes "my usage example did not work because you are on the wrong version of the library/using the wrong SDK…

If a library isn't widely used (and is small enough) you can paste the entire thing into the context to ensure the LLM can use it effectively.
Post reply on HN