Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

danluu.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#31

Post by mplanchard »

IsTom wrote: post #6
stephantul wrote: post #3

The (relevant) segue into tires elevates this post so much. I don’t understand why, but it does

I'm a little bit confused about these tire claims as > down to 0C / 32 F (he didn't test colder conditions) I get that people live in different places, but that's a huge caveat. How's that winter if you're not below 0°C? That sounds like "winter tires are worse than all-seasons tires in winter if you exclude winter".

Yes, this stuck out to me, too. I use winter tires in the winter specifically for the snow and consistently below-freezing temps. Are there places people bother with winter tires that aren’t actually cold in the winter?

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#32

Post by menaerus »

jbellis wrote: post #18

While I'm generally sympathetic to the idea that the public coding benchmarks are inadequate, to the point that I've written my own in the past and will likely do so again, the complaint here is that "these tasks don't match what I do in my day job" which is ~always going to be the case. The hope with benchmarks is that you can capture properties that generalize, from examining performance against small set of tasks,…

I started using gemini with caution given the "much worse" benchmarking points it has gotten and still does but in practice there's very little evidence I found in comparison to claude models. It performs really well on non trivial tasks.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#33

Post by menaerus »

mplanchard wrote: post #31
IsTom wrote: post #6

Earlier quoted context omitted.

I'm a little bit confused about these tire claims as > down to 0C / 32 F (he didn't test colder conditions) I get that people live in different places, but that's a huge caveat. How's that winter if you're not below 0°C? That sounds like "winter tires are worse than all-seasons tires in winter if you exclude winter".

Yes, this stuck out to me, too. I use winter tires in the winter specifically for the snow and consistently below-freezing temps. Are there places people bother with winter tires that aren’t actually cold in the winter?

If warranted by the law then yes, you have to abide to it.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#34

Post by mplanchard »

menaerus wrote: post #33
mplanchard wrote: post #31

Earlier quoted context omitted.

Yes, this stuck out to me, too. I use winter tires in the winter specifically for the snow and consistently below-freezing temps. Are there places people bother with winter tires that aren’t actually cold in the winter?

If warranted by the law then yes, you have to abide to it.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#35

Post by 27183 »

threetonesun wrote: post #28
27183 wrote: post #25

Earlier quoted context omitted.

Lately I've just been leaving my snow tires on year round (Bridgestone Blizzaks). It gets cold here, between November and April the number of days with daytime high above freezing is small. I used to run all seasons in the summer and snows in the winter but they were lasting too long that way. It's better to wear them out within ~5yr.

I've done that with "mild" Winter tires, really the thing full Summer tires excell at is removing water, which requires a center tread that's useless in snow. So if you don't get a lot of rain and it doesn't get too hot (some Winter compounds will get mushy and wear very quickly) they're fine. But really I've just described a Winter rated All-Season at this point, and you should buy those.

All season tires are useless on snow and ice unless they're brand new. They're OK the first winter, after that it's bad news. On heavier vehicles I run BFGoodrich All Terrain T/A tires year round. They have good siped treads which grip on ice. For the car, studless Blizzaks have been holding up well year round. No abnormal wear so far, and they do fine in the wet, dry, heat, etc. I probably only drive 1-2 days/yr over 90°F though, in a hotter climate they might not be so great.

[edit] obviously this is a tradeoff--I'm trading slightly reduced hot/dry/wet performance for massively increased winter performance. The reduction in summer performance is small enough to not be noticeable, whereas the increase in winter performance is large. On all season tires I would have to chain up a dozen or so times per year, often just to move the car like 3 car lengths out of a parking spot. I've only ever had to put the chains on once with snow tires on the car, and that was bashing through 6" of unplowed crusty icy stuff up a steep driveway.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#36

Post by menaerus »

mplanchard wrote: post #34
menaerus wrote: post #33

Earlier quoted context omitted.

If warranted by the law then yes, you have to abide to it.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

Sometimes the winters may be cold, some other times they may be exceptionally warm, and sometimes they're a mix of both. The law remains the same regardless what the winter was or is like. Countries that have exceptionally warm winters do not really have a winter conditions so I'd guess their law wouldn't have a requirement for winter tires or chains

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#37

Post by auxym »

mplanchard wrote: post #34
menaerus wrote: post #33

Earlier quoted context omitted.

If warranted by the law then yes, you have to abide to it.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

Quebec, Canada mandates winter tires between December 1st and March 15th.

(But the winters are cold).

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#38

Post by WarmWash »

jdw64 wrote: post #7

I feel like AI development is hitting a wall now. Evaluations are driven by benchmarks, but I have no idea who is even evaluating the validity of these benchmarks. The idea that continuously training and scaling models will automatically yield broad capabilities across multiple domains hardly seems true anymore. Benchmarks like Senior SWE-Bench apply discontinuous and arbitrary thresholds to evaluate results. If the…

Math produces a bunch of theories when you extrapolate a system forward (or sometimes backwards)

Experimentation is the hammer that smashes all the incorrect theories.

Without the ability to do experimentation, deriving new laws is virtually impossible. A new next step for AI would be coming up with novel experiments, because that is often the hardest part.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#40

Post by lh712 »

cb321 wrote: post #24
lh712 wrote: post #16

Earlier quoted context omitted.

Well, exactly, and I think that that is in line with what I originally wrote, and which is that you need the general relativity principle (by general I don't mean the general relativity but the concept of the equivalence of inertial frames) and the theory electromagnetism. (By the way, I am familiar with the Landau--Lifshitz textbook.) (1) The relativity principle is exactly what you refer to yourself. You can't appl…

Yeah. My point was mostly to expand upon your original post - sorry if it sounded like I disagreed. You say the same in your original "iteration of speculation and observation". Everything else is about what "derive" and "knowledge" might mean (I agree "medieval" usually means pre-Renaissance and Galileo is modern-era), how much empirical "proof" is in "derive", etc. However, all you need for "possible" is "the idea"…

Thank you for your reply! Yes, I fully agree that meaning of "derive" and other related concepts are non-trivial. (And I actually changed my own "working definition" between my two posts; while in the first one I was leaning more towards theories anchored by observation, even if indirectly, in the second one I slipped into more mathematical/philosophical perspective. The second part of my first reply makes sense only from the first kind of perspective, because in that part I imagine that the observed reality, even at the medieval level of knowledge, might perhaps necessitate a particular kind of underlying theory, from which the relativity principle would then "fall out". That was the only way I could think of at that moment how SR could have been "derived" without further observations.)

Your points about Leibniz are very interesting, and while I find that line of thought fascinating I must admit my own lack of knowledge of the historical context. In any case, thank you for pointing out that idea!

Post reply on HN