Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3.7 Sonnet and Claude Code

anthropic.com

Post reply on HN
Print view View original

751–760 of 1001 posts

Re: Claude 3.7 Sonnet and Claude Code

#751

Post by ZeroTalent »

throwaway454647 wrote: post #725
babyshake wrote: post #190

Earlier quoted context omitted.

One thing I would love to have fixed - I type in a prompt, the model produces 90% or even 100% of the answer, and then shows an error that the system is at capacity and can't produce an answer. And then the response that has already been provided is removed! Please just make it where I can still have access to the response that has been provided, even if it is incomplete.

I'll be publishing a Firefox extension as a temporary fix, will post it here. (I don't use Chrome.)

I think tampermonkey code is a better solution?

Re: Claude 3.7 Sonnet and Claude Code

#752

Post by jonplackett »

nightpool wrote: post #542
anotherpaulg wrote: post #331

Claude 3.7 Sonnet scored 60.4% on the aider polyglot leaderboard [0], WITHOUT USING THINKING. Tied for 3rd place with o3-mini-high. Sonnet 3.7 has the highest non-thinking score, taking that title from Sonnet 3.5. Aider 0.75.0 is out with support for 3.7 Sonnet [1]. Thinking support and thinking benchmark results coming soon. [0] https://aider.chat/docs/leaderboards/ [1] https://aider.chat/HISTORY.html#aider-v0750

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

I like to make up my own tests, that way you know it is actually thinking.

Tests that require thinking about the physical world are the most revealing.

My new favourite is:

You have 2 minutes to cool down a cup of coffee to the lowest temp you can.

You have two options: 1. Add cold milk immediately, then let it sit for 2 mins.

2. Let it sit for 2 mins, then add cold milk.

Which one cools the coffee to the lowest temperature and why?

Phrased this way without any help, all but the thinking models get it wrong

Re: Claude 3.7 Sonnet and Claude Code

#753

Post by wokwokwok »

M4v3R wrote: post #710
wokwokwok wrote: post #665

Earlier quoted context omitted.

"no lines of code directly written, just directing the AI" /skeptical face. Without fail, every. single. person. I've met who says that, actually means "except for the code that I write", or "except for how I link the code it build together by hand". If you are 50kloc in to a large complex project that you have literally written none of, and have, eg. used cursor to generate the code without any assistance... well, y…

I’m currently doing something very similar to what GP is doing - I’m building a hobby project that’s a desktop app with web frontend. It’s a map editor with a 3D view. My estimate is that 80-90% of the code was written by AI. Sure, I did have to intervene or write some more complex parts myself but it’s still exciting to me that in many cases it took just a single prompt to add a new feature to it or change existing…

> My estimate is that 80-90% of the code was written by AI

Nice! It is entirely reasonable both to do that and to be excited about it.

…buuut, if that’s what you’re doing, you should say so.

Not:

“no lines of code directly written, just directing the AI”

Because those (gluing together AI code by hand and having the agent do everything) are different things, and one of them is much much MUCH harder to get right than the other one.

That last 10-15%. Self driving cars are the same story right?

Re: Claude 3.7 Sonnet and Claude Code

#754

Post by [deleted] »

Marazan wrote: post #717
anotherpaulg wrote: post #651

Earlier quoted context omitted.

I try not to let perfect be the enemy of good. All benchmarks have limitations. The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over…

Having the verbatim answer to the test is not a "limitation" it is an invalidation.

[deleted]

Re: Claude 3.7 Sonnet and Claude Code

#755

Post by jonplackett »

khafra wrote: post #698
mikae1 wrote: post #686

Earlier quoted context omitted.

It's clear that progress is incremental at this point. At the same time Anthropic and OpenAI are bleeding money. It's unclear to me how they'll shift to making money while providing almost no enhanced value.

Yudkowsky just mentioned that even if LLM progress stopped right here, right now, there are enough fundamental economic changes to provide us a really weird decade. Even with no moat, if the labs are in any way placed to capture a little of the value they've created, they could make high multiples of their investors' money.

Yep totally agree. It will also depend who captures the most eyeballs.

ChatGPT is already my default first place to check something, where it was Google for the previous 20+ years.

Re: Claude 3.7 Sonnet and Claude Code

#756

Post by berkes »

datadeft wrote: post #749

I am not sure how good these Exercism tasks are for measuring how good at a model with coding. My experience is that these models could write a simple function and get it right if it does not require any out of the box thinking (so essentially offloading the boilerplate part of coding). When it comes to think creatively and have a much better solution to a specific task that would require the think 2-3 steps ahead th…

I think many of the "AI can do coding" narratives don't see what coding means in real situations.

It's finding out why "jbdoe1337" added this large if/else around the entire function body back in 2016 - it seems important business logic, but the commit just says "updated code". And how the h*ll this interaction between the conf.ini files, the conf/something.json and the ENV vars works. Why sometimes the ENV var overrides a value in the ini and why its sometimes the other way around. But also finding that when you clean it up, everything falls apart.

It's discussing with the stakeholders why "adding a delete button" isn't as easy as just putting a button there, but that it means designing a whole cascading deletion strategy and/or trashcan and/or soft-delete and/or garbage-collection.

It's finding out why - again - the grumb pipeline crashes with the typebar checker, when used through mpm-yearn package manager. Both in containers and on a osx machine but not on Linux Brobuntu 22.12 LTLS.

It's moving stuff in the right abstraction layer. It's removing abstractions while introducing others. KISS vs future flexibility. It's gut feeling when to apply DRY and when to embrace it.

And then, if your lucky, churning out boilerplate or new code for 120 minutes a week.

I'm glad that this 120 minutes can be improved with AI and become 20 minutes. Truly. But this is not what (senior?) programmers do. Despite what the hyped up AI press makes us believe. It only shows they have no idea what the "real" problems and time-consumers are for programmers.

Re: Claude 3.7 Sonnet and Claude Code

#757

Post by Amekedl »

khafra wrote: post #698
mikae1 wrote: post #686

Earlier quoted context omitted.

It's clear that progress is incremental at this point. At the same time Anthropic and OpenAI are bleeding money. It's unclear to me how they'll shift to making money while providing almost no enhanced value.

Yudkowsky just mentioned that even if LLM progress stopped right here, right now, there are enough fundamental economic changes to provide us a really weird decade. Even with no moat, if the labs are in any way placed to capture a little of the value they've created, they could make high multiples of their investors' money.

Oh really? How are these changes supposed to look like? Who will pay up essentially? I don't really see it, aside from the m$ business case of offering AI as a guise for violating privacy much harsher to better sell ads.

Re: Claude 3.7 Sonnet and Claude Code

#758

Post by jahooma »

Daniel_Van_Zant wrote: post #474

Being able to control how many tokens are spent on thinking is a game-changer. I've been building fairly complex, efficient, systems with many LLMs. Despite the advantages, reasoning models have been a no-go due to how variable the cost is, and how hard that makes it to calculate a final per-query cost for the customer. Being able to say "I know this model can always solve this problem in this many thinking tokens" a…

Yup, it's just what we wanted for our coding agent. Codebuff can enter a "Deep thinking" mode and we can tell it to burn a lot of tokens hahaha.

Re: Claude 3.7 Sonnet and Claude Code

#759

Post by fragmede »

koakuma-chan wrote: post #297
trekkie1024 wrote: post #275

Earlier quoted context omitted.

Could you share an example?

TLDR: told it to implement a grid view as an alternative to the existing list view, and specifically told it to DRY the code. What it did? Copy and pasted the list view implementation (definitely not DRY), and tried to make it a grid, and even though it is a grid, it looks terrible ( https://i.imgur.com/fJiSjq4.png ). I don't understand how people use cursor and all that other shit when it cannot follow such simple i…

Points for being specific with it's shortcomings! Expecting it to get it in one shot isn't how to work best with them. It takes some cajoling and at this point in time, sometimes it is still faster to do it the old fashioned way of you already know what you're doing.

Re: Claude 3.7 Sonnet and Claude Code

#760

Post by akoboldfrying »

jonplackett wrote: post #752
nightpool wrote: post #542

Earlier quoted context omitted.

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temp…

The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet too.

I think a better test of thinking is to provide detail about something so mundane and esoteric that no one would have ever thought to communicate it to other people for entertainment, and then ask it a question about that pile of boring details.

Post reply on HN