Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3.7 Sonnet and Claude Code

anthropic.com

Post reply on HN
Print view View original

761–770 of 1001 posts

Re: Claude 3.7 Sonnet and Claude Code

#761

Post by chvid »

chvid wrote: post #712
nightpool wrote: post #542

Earlier quoted context omitted.

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

They leak the second they are used on a model behind an API, don't they?

As far as I can tell the only way of doing a comparison of two models, that cannot be easily gamed, is being having them in open weights form and then running them against a benchmark that was created after both of the two models were created.

Re: Claude 3.7 Sonnet and Claude Code

#762

Post by xx_ns »

akoboldfrying wrote: post #760
jonplackett wrote: post #752

Earlier quoted context omitted.

I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temp…

The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet…

Out of curiosity, what is the answer? From your comment, it seems like the more obvious choice is the incorrect one.

EDIT: By the more obvious one, I mean letting it cool and then adding milk. As the temperature difference between the coffee and the surrounding air is higher, the coffee cools down faster. Is this wrong?

Re: Claude 3.7 Sonnet and Claude Code

#764

Post by danbruc »

jonplackett wrote: post #752
nightpool wrote: post #542

Earlier quoted context omitted.

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temp…

No need for thinking, that question can be found discussed and explained many times online and has almost certainly been part of the training data.

Re: Claude 3.7 Sonnet and Claude Code

#765

Post by j_maffe »

jmehman wrote: post #705
j_maffe wrote: post #472

Earlier quoted context omitted.

Do you have any specific tips for the last point? I completely agree with it and have set up a fairly robust Obsidian note taking structure that will benefit greatly from an agentic assistant. Do you use specific tools or workframe for this?

You know about the copilot plugin for obsidian?

Yes I've started using it but it feels significantly underdevoloped compared to GH Copilot or Cursor. I've considered opening the vault in VSC actually.

Re: Claude 3.7 Sonnet and Claude Code

#766

Post by s-macke »

akoboldfrying wrote: post #760
jonplackett wrote: post #752

Earlier quoted context omitted.

I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temp…

The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet…

Simple Bench goes in this direction: https://simple-bench.com/

Re: Claude 3.7 Sonnet and Claude Code

#767

Post by danbruc »

xx_ns wrote: post #762
akoboldfrying wrote: post #760

Earlier quoted context omitted.

The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet…

Out of curiosity, what is the answer? From your comment, it seems like the more obvious choice is the incorrect one. EDIT: By the more obvious one, I mean letting it cool and then adding milk. As the temperature difference between the coffee and the surrounding air is higher, the coffee cools down faster. Is this wrong?

That is the correct answer. Also there is a lot of potential nuance, like evaporation or when you take the milk out of the fridge or the specific temperatures of everything, but under realistic settings adding the milk late will get you the colder coffee.

Re: Claude 3.7 Sonnet and Claude Code

#768

Post by Zamicol »

e12e wrote: post #632
hubraumhugo wrote: post #131

You can get your HN profile analyzed by it and it's pretty funny :) https://hn-wrapped.kadoa.com/ I'm using this to test the humor of new models.

Thanks for sharing - I really feel Claude gets me ;-) https://hn-wrapped.kadoa.com/e12e?share > Your comments read like Warren and Brandeis met Alan Kay at a Norwegian tech conference. I consider this high praise indeed, lol.

> One of your comments about the absurdity of centralized authentication will spark a 300+ comment thread and lead to a new open standard for federated identity.

Hmmm...

Re: Claude 3.7 Sonnet and Claude Code

#769

Post by Zamicol »

jjice wrote: post #279
hubraumhugo wrote: post #131

You can get your HN profile analyzed by it and it's pretty funny :) https://hn-wrapped.kadoa.com/ I'm using this to test the humor of new models.

This is absolutely hilarious! Thanks for posting. It feels weighted towards some specific things (I assume this is done by the LLM caring about later context more?) - making it debatably even funnier. > You're the only person who gets excited about trailing commas in SQL. Even the database administrators are like 'dude, it's just a comma.'

There's dozens of us!

Re: Claude 3.7 Sonnet and Claude Code

#770

Post by AlfeG »

sizzle wrote: post #735
AlfeG wrote: post #729

Ahha, recently my daugher come to me with 3rd grade math problem. "Without rearranging the digits 1 2 3 4 5, insert mathematical operation signs and, if necessary, parentheses between them so that the resulting expression equals 40 and 80. The key is that you can combine digits (like 12+3/45) but you cannot change their order from the original sequence 1,2,3,4,5" Grok3, Claude, Deepseek, Qwen all failed to solve this…

This is what they are expecting 3rd graders to solve in math? Pretty hard for that age?

They have a lessons before on order of the expressions and some similiar problems. They were able to solve for 80, but stuck on 40 and asked me.
Post reply on HN