Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3.7 Sonnet and Claude Code

anthropic.com

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: Claude 3.7 Sonnet and Claude Code

#711

Post by fixprix »

wokwokwok wrote: post #665
d_watt wrote: post #114

I'm about 50kloc into a project making a react native app / golang backend for recipes with grocery lists, collaborative editing, household sharing, so a complex data model and runtime. Purely from the experiment of "what's it like to build with AI, no lines of code directly written, just directing the AI." As I go through features, I'm comparing a matrix of Cursor, Cline, and Roo, with the various models. While I'm…

"no lines of code directly written, just directing the AI" /skeptical face. Without fail, every. single. person. I've met who says that, actually means "except for the code that I write", or "except for how I link the code it build together by hand". If you are 50kloc in to a large complex project that you have literally written none of, and have, eg. used cursor to generate the code without any assistance... well, y…

If you know how to architect code well, you can guide the AI to create smaller more targeted modules. That way as you 'write code with AI', you give it a targeted subset of the files to edit on each prompt.

In a way the AI becomes the dev and you become the code reviewer. Often as the AI is writing the code, you're thinking about the next step.

Re: Claude 3.7 Sonnet and Claude Code

#712

Post by chvid »

nightpool wrote: post #542
anotherpaulg wrote: post #331

Claude 3.7 Sonnet scored 60.4% on the aider polyglot leaderboard [0], WITHOUT USING THINKING. Tied for 3rd place with o3-mini-high. Sonnet 3.7 has the highest non-thinking score, taking that title from Sonnet 3.5. Aider 0.75.0 is out with support for 3.7 Sonnet [1]. Thinking support and thinking benchmark results coming soon. [0] https://aider.chat/docs/leaderboards/ [1] https://aider.chat/HISTORY.html#aider-v0750

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

They leak the second they are used on a model behind an API, don't they?

Re: Claude 3.7 Sonnet and Claude Code

#713

Post by jrflowers »

anotherpaulg wrote: post #651
nightpool wrote: post #542

Earlier quoted context omitted.

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

I try not to let perfect be the enemy of good. All benchmarks have limitations. The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over…

>I try not to let perfect be the enemy of good. All benchmarks have limitations.

Overfitting is one of the fundamental issues to contend with when trying to figure out if any type of model at all is useful. If your leaderboard corresponds to vibes and that is your target, you could just have a vibes leaderboard

Re: Claude 3.7 Sonnet and Claude Code

#714

Post by M4v3R »

tkgally wrote: post #575

In early January, inspired by a post by Simon Willison, I had Claude 3.5 Sonnet write a couple of stand-up comedy routines as done by an AI chatbot speaking to a mixed audience of AIs and humans. I thought the results were pretty good—the only AI-produced humor that I had found even a bit funny. I tried the same prompt again just now with Claude 3.7 Sonnet in thinking mode, and I found myself laughing more than I did…

Wow, that was… Surprisingly good. I did laugh a few times and I really didn’t expect to.

Re: Claude 3.7 Sonnet and Claude Code

#715

Post by re-thc »

Ballas wrote: post #678
tw1984 wrote: post #664

Earlier quoted context omitted.

is there any public info on why such DeepSeek R1 + claude-3-5 combo worked better than using a single model?

From my experiments with the Deepseek Qwen-32b distill model, the Deepseek model did not follow the edit instructions - the format was wrong. I know the distill models are not at all the same as the full model, but that could provide a clue. Combine that information with the scores, then you have a reasonable hypothesis.

> I know the distill models are not at all the same as the full model

It's far worse than that. It's not the model (Deepseek) at all. It's Qwen enhanced with Deepseek. So it's Qwen still.

Re: Claude 3.7 Sonnet and Claude Code

#716

Post by re-thc »

rudedogg wrote: post #680
Sterling9x wrote: post #675

Earlier quoted context omitted.

That's a file context problem because you use cursor or cline or some other crap context maker. Try Clood. Unless "anthropic high usage" which I just watch the incident reports I one shot features regularly. At a high skill level. Not front end. Back end c# in a small but great framework that has poor documentation. Not just endpoints but full on task queues. So really, it's a context problem. You're just not laser f…

Wtf is “clood”?

It's cloud that takes "o"ver "u"

Re: Claude 3.7 Sonnet and Claude Code

#717

Post by Marazan »

anotherpaulg wrote: post #651
nightpool wrote: post #542

Earlier quoted context omitted.

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

I try not to let perfect be the enemy of good. All benchmarks have limitations. The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over…

Having the verbatim answer to the test is not a "limitation" it is an invalidation.

Re: Claude 3.7 Sonnet and Claude Code

#720

Post by kiraaa »

bcherny wrote: post #91

Hi everyone! Boris from the Claude Code team here. @eschluntz, @catherinewu, @wolffiex, @bdr and I will be around for the next hour or so and we'll do our best to answer your questions about the product.

when there are two commands in a prompt example

do A and then do B.

the model completely ignores the second task B.

Post reply on HN