771–780 of 1001 posts
You can get your HN profile analyzed by it and it's pretty funny :) https://hn-wrapped.kadoa.com/ I'm using this to test the humor of new models.
You can get your HN profile analyzed by it and it's pretty funny :) https://hn-wrapped.kadoa.com/ I'm using this to test the humor of new models.
Oh wow the predictions are surprisingly good. It predicted exactly what I'm working on right now despite never having revealed that anywhere The roast could probably be improved. Mine wasn't offensive at all.
I'm curious!
Earlier quoted context omitted.
> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?
I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temp…
I C&P'd it into Claude 3.7 with thinking, and it gave the correct answer (which I'm pretty sure is #2). Including the CoT, where it actually does math (which I haven't checked), and final response.
# THINKING
Let's analyze the two options.
Option 1: Add cold milk immediately, then let it sit for 2 mins.
Option 2: Let it sit for 2 mins, then add cold milk.
To determine which option leads to the lowest final temperature, I need to consider the physics of cooling.
First, let's understand how cooling works in these scenarios:
1. When a hot liquid sits, it cools according to Newton's Law of Cooling, which states that the rate of heat loss is proportional to the temperature difference between the object and its surroundings.
- The rate of cooling is faster when the temperature difference is greater
- As the liquid cools, the rate of cooling decreases
2. When we add cold milk to hot coffee, we get an immediate temperature drop due to mixing liquids of different temperatures. The final temperature after mixing can be calculated using the principle of thermal equilibrium.
Now, let's analyze each option:
Option 1: Add cold milk immediately, then let it sit for 2 mins
- The coffee temperature drops immediately when milk is added
- Then this coffee+milk mixture cools for 2 minutes
- The cooling rate will be based on the temperature difference between the coffee+milk mixture and room temperature
Option 2: Let it sit for 2 mins, then add cold milk
- The black coffee cools for 2 minutes
- Then milk is added, causing an immediate drop in temperature
- The cooling rate during the 2 minutes will be based on the temperature difference between the black coffee and room temperature
The key insight is that hot coffee cools faster than warm coffee because the rate of cooling is proportional to the temperature difference. Since black coffee starts at a higher temperature than coffee with milk, it will lose more heat energy during the 2-minute cooling period.
Let's make this more concrete with some theoretical numbers:
- Let's say black coffee starts at 90°C
- Room temperature is 20°C
- Cold milk is at 5°C and makes up 10% of the final volume
Option 1:
- Adding milk immediately might bring the temperature to about 81.5°C (90 × 0.9 + 5 × 0.1)
- This mixture then cools for 2 minutes from 81.5°C to some lower temperature, let's call it T1
Option 2:
- The black coffee cools for 2 minutes from 90°C to some temperature, let's call it T2
- Then we add milk, which further reduces the temperature to 0.9 × T2 + 0.1 × 5
The question is: which is lower, T1 or 0.9 × T2 + 0.1 × 5?
According to Newton's Law of Cooling, the black coffee in Option 2 will lose more heat during the 2 minutes because it starts at a higher temperature and thus has a greater temperature difference with the surroundings. This means the drop from 90°C to T2 will be greater than the drop from 81.5°C to T1.
Let's think about this mathematically:
According to Newton's Law of Cooling: T(t) = Ts + (T0 - Ts) × e^(-kt) Where:
- T(t) is the temperature at time t
- Ts is the surrounding temperature
- T0 is the initial temperature
- k is the cooling constant
For Option 1:
T1 = 20 + (81.5 - 20) × e^(-k × 2) = 20 + 61.5 × e^(-2k)
For Option 2, before adding milk:
T2 = 20 + (90 - 20) × e^(-k × 2) = 20 + 70 × e^(-2k)
After adding milk in Option 2, the temperature would be:
0.9 × T2 + 0.1 × 5 = 0.9 × (20 + 70 × e^(-2k)) + 0.5
= 18 + 63 × e^(-2k) + 0.5
= 18.5 + 63 × e^(-2k)
Now we compare:
T1 = 20 + 61.5 × e^(-2k)
Final temp in Option 2 = 18.5 + 63 × e^(-2k)
For any positive value of k (which it must be for cooling), e^(-2k) is between 0 and 1.
If e^(-2k) = 1 (no cooling at all):
T1 = 20 + 61.5 = 81.5
Final temp in Option 2 = 18.5 + 63 = 81.5 (They would be the same)
If e^(-2k) = 0 (instant cooling to room temperature): T1 = 20 Final temp in Option 2 = 18.5 (Option 2 would be better)
For values in between, we need to compare 20 + 61.5 × e^(-2k) vs 18.5 + 63 × e^(-2k)
Subtracting: (T1 - Final temp in Option 2) = 1.5 - 1.5 × e^(-2k)
This will be positive (meaning T1 is higher) when e^(-2k) So, mathematically, Option 2 (letting it sit, then adding milk) results in a lower final temperature.
This makes physical sense because:
1. The hotter black coffee loses more heat during the 2 minutes than the cooler coffee+milk mixture would
2. The cold milk provides the same cooling effect whether added at the beginning or end
Therefore, Option 2 (letting the coffee sit for 2 minutes, then adding cold milk) will result in the lowest final temperature.
# ANSWER
# Coffee Cooling Analysis
## Option 2 wins: Let it sit for 2 mins, then add cold milk
This will give you the lowest final temperature, and here's why:
### The physics behind it:
1. *Newton's Law of Cooling*: The rate at which something cools is proportional to the temperature difference between the object and its surroundings.
2. *Key insight*: Hotter objects cool faster than cooler ones because the temperature gap with the environment is larger.
### Comparing the options:
*Option 1 (Add milk first, then wait):* - Milk immediately lowers coffee temperature - The coffee+milk mixture cools for 2 minutes, but at a slower rate because it's already cooler
*Option 2 (Wait 2 mins, then add milk):* - Black coffee starts hotter and cools faster during the 2 minutes - More total heat is lost during the waiting period - Then milk provides the same temperature drop at the end
The black coffee's higher initial temperature difference with the room causes it to lose more total heat energy during the waiting period, making Option 2 the more effective cooling strategy.
Earlier quoted context omitted.
It's clear that progress is incremental at this point. At the same time Anthropic and OpenAI are bleeding money. It's unclear to me how they'll shift to making money while providing almost no enhanced value.
Yudkowsky just mentioned that even if LLM progress stopped right here, right now, there are enough fundamental economic changes to provide us a really weird decade. Even with no moat, if the labs are in any way placed to capture a little of the value they've created, they could make high multiples of their investors' money.
Claude 3.7 Sonnet scored 60.4% on the aider polyglot leaderboard [0], WITHOUT USING THINKING. Tied for 3rd place with o3-mini-high. Sonnet 3.7 has the highest non-thinking score, taking that title from Sonnet 3.5. Aider 0.75.0 is out with support for 3.7 Sonnet [1]. Thinking support and thinking benchmark results coming soon. [0] https://aider.chat/docs/leaderboards/ [1] https://aider.chat/HISTORY.html#aider-v0750
Earlier quoted context omitted.
> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?
I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temp…
Earlier quoted context omitted.
> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?
I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temp…
You just ruined your own test by publishing it on the internets
Earlier quoted context omitted.
> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?
I try not to let perfect be the enemy of good. All benchmarks have limitations. The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over…
The Aider Polyglot website also states that the benchmark " ...asks the LLM to edit source files to complete 225 coding exercises".
However, when looking at the actual tests [0], it is not about editing code bases, it's rather just solving simple programming exercies? What am I missing?