Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3.7 Sonnet and Claude Code

anthropic.com

Post reply on HN
Print view View original

651–660 of 1001 posts

Re: Claude 3.7 Sonnet and Claude Code

#651

Post by anotherpaulg »

nightpool wrote: post #542
anotherpaulg wrote: post #331

Claude 3.7 Sonnet scored 60.4% on the aider polyglot leaderboard [0], WITHOUT USING THINKING. Tied for 3rd place with o3-mini-high. Sonnet 3.7 has the highest non-thinking score, taking that title from Sonnet 3.5. Aider 0.75.0 is out with support for 3.7 Sonnet [1]. Thinking support and thinking benchmark results coming soon. [0] https://aider.chat/docs/leaderboards/ [1] https://aider.chat/HISTORY.html#aider-v0750

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

I try not to let perfect be the enemy of good. All benchmarks have limitations.

The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over the last 18+ months.

To be clear, the purpose of the benchmark is to help me quantitatively assess and improve aider and make it more effective. But it's also turned out to be a great way to measure the coding skill of LLMs.

Re: Claude 3.7 Sonnet and Claude Code

#652

Post by MyOutfitIsVague »

sejje wrote: post #647
doctoboggan wrote: post #636

Earlier quoted context omitted.

I like Aider but I've turned off auto-commit. I just can't seem to let the AI actually commit code for me. Do you regularly let Aider commit for you? How much do you review the code written by it?

I don't let it auto commit, either. I don't like committing in a broken state, and the llm breaks things plenty often.

What's wrong with committing in a broken state if you squash those into a working state before pushing?

Re: Claude 3.7 Sonnet and Claude Code

#653

Post by whamlastxmas »

ANewFormation wrote: post #159
hubraumhugo wrote: post #131

You can get your HN profile analyzed by it and it's pretty funny :) https://hn-wrapped.kadoa.com/ I'm using this to test the humor of new models.

Oh god that's genuinely way more amusing than I thought llm systems were capable of.

This is actually by far the best example of humor by an LLM I’ve ever seen

Re: Claude 3.7 Sonnet and Claude Code

#654

Post by sangnoir »

ipsum2 wrote: post #606
sangnoir wrote: post #394

Earlier quoted context omitted.

I'm not affiliated with Anthropic, but it seems like doing this will commoditize Claude (the AIaaS). Hosted AI providers are doing all they can to move away from being interchangeable commodities; it's not good for Anthropic's revenue for users to be able to easily swap-out the backend of Cloud Code to a local Olama backend, or a cheaper hosted DeepSeek. Open sourcing Claude Code would make this option 1 or 2 forks/P…

It's not hard to make, its a relatively simple CLI tool so there's no moat. Also, the minified source code is available.

> It's not hard to make, its a relatively simple CLI tool so there's no moat

There are similar open source CLI tools that predate Claude Coder. Its reasonable to assume Anthropic chose not to contribute to those projects for reasons other than complexity, and charitably Anthropic likely plans for differentiating features.

> Also, the minified source code is available

The redistribution license - or lack thereof - will be the stumbling block to directly reusing code authored by Anthropic without authorization.

Re: Claude 3.7 Sonnet and Claude Code

#655

Post by pzo »

pclmulqdq wrote: post #604
anotherpaulg wrote: post #580

Earlier quoted context omitted.

Using up to 32k thinking tokens, Sonnet 3.7 set SOTA with a 64.9% score. 65% Sonnet 3.7, 32k thinking 64% R1+Sonnet 3.5 62% o1 high 60% Sonnet 3.7, no thinking 60% o3-mini high 57% R1 52% Sonnet 3.5

Also for $36.83 compared to o1's $186.50

But also for $36.83 compared to DeepSeek R1 + claude-3-5 it's $13.29 and for latter "Percent using correct edit format" is 100% vs 97.8% for 3.7.

edit: would be interesting to see how combo DeepSeek R1 + claude-3-7 performs.

Re: Claude 3.7 Sonnet and Claude Code

#656

Post by TechDebtDevin »

anjel wrote: post #29
bnc319 wrote: post #2

Pretty amazing how DeepSeek started the visual reasoning trend, xAI featured it in their latest release, and now Anthropic does the same.

I took DS visual reasoning to be an elegant misdirect from how much slower DS returns your query's output.

You almost have to do this, or atleast some sort of progress bar, else people will think their requests failed and spam the server.

Re: Claude 3.7 Sonnet and Claude Code

#657

Post by aizk »

Ninjinka wrote: post #201
bcherny wrote: post #91

Hi everyone! Boris from the Claude Code team here. @eschluntz, @catherinewu, @wolffiex, @bdr and I will be around for the next hour or so and we'll do our best to answer your questions about the product.

How is your largest customer, Cursor, taking the news that you'll be competing directly with them?

Anthropic is still making the shovels

Re: Claude 3.7 Sonnet and Claude Code

#658

Post by aizk »

whynotminot wrote: post #487
fallinditch wrote: post #171

Earlier quoted context omitted.

My guess is that, yes, the software development job market is being massively disrupted, but there are things you can do to come out on top: * Learn more of the entire stack, especially the backend, and devops. * Embrace the increased productivity on offer to ship more products, solo projects, etc * Be highly selective as far as possible in how you spend your productive time: being uber-effective can mean thinking an…

> Learn more of the entire stack, especially the backend, and devops. I actually wonder about this. Is it better to gain some relatively mediocre experience at lots of things? AI seems to be pretty good at lots of things. Or would it be better to develop deep expertise in a few things? Areas where even smart AI with reasoning still can get tripped up. Trying to broaden your base of expertise seems like it’s always a…

I was advised to be T shaped, wide reach + one narrow domain you can really nail.

Re: Claude 3.7 Sonnet and Claude Code

#659

Post by alexjplant »

hubraumhugo wrote: post #131

You can get your HN profile analyzed by it and it's pretty funny :) https://hn-wrapped.kadoa.com/ I'm using this to test the humor of new models.

In my case the first two roasts were contrivances but the last one carries the bunch:

> For someone who claims to be only 33, you have the technological opinions of at least three 60-year-old UNIX greybeards stacked in a trenchcoat.

Guilty as charged :-3

Re: Claude 3.7 Sonnet and Claude Code

#660

Post by Imustaskforhelp »

babyshake wrote: post #190
bcherny wrote: post #91

Hi everyone! Boris from the Claude Code team here. @eschluntz, @catherinewu, @wolffiex, @bdr and I will be around for the next hour or so and we'll do our best to answer your questions about the product.

One thing I would love to have fixed - I type in a prompt, the model produces 90% or even 100% of the answer, and then shows an error that the system is at capacity and can't produce an answer. And then the response that has already been provided is removed! Please just make it where I can still have access to the response that has been provided, even if it is incomplete.

Yup. Its a great issue which messes like , cmon you were there at the last line.
Post reply on HN