Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires • Hacker News Board

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#41

Post by pm215 » Mon, Sep 14, 2026, 3:41 PM UTC

rjsw wrote: post #29

IsTom wrote: post #17

Earlier quoted context omitted.

> but I guess you get into problems with it being ice and snow that you're testing If it's not actively snowing for a few days roads get clear by thawing during day because of salt/cars being hot/sun shining. The "summer tyres get too hard in the cold" idea could very well come from places where the typical winter is at least < -5°C. It's not uncommon in central/eastern/northern europe for temperatures to be < -10°C…

The crossover point where winter tyres work better is < +7°C, it doesn't need to be below freezing.

The linked article claims "in dry conditions, summer tires have the best grip down to 0C", though, which doesn't seem to match where you suggest the crossover point is.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#42

Post by mplanchard » Mon, Sep 14, 2026, 3:54 PM UTC

auxym wrote: post #37

mplanchard wrote: post #34

Earlier quoted context omitted.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

Quebec, Canada mandates winter tires between December 1st and March 15th. (But the winters are cold).

Didn’t know it was required, but it makes sense, as it is indeed quite cold. Good thing I always have winter tires on in the winter anyway, so I’m not breaking the law when I drive to Montreal!

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#43

Post by mplanchard » Mon, Sep 14, 2026, 3:59 PM UTC

menaerus wrote: post #36

mplanchard wrote: post #34

Earlier quoted context omitted.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

Sometimes the winters may be cold, some other times they may be exceptionally warm, and sometimes they're a mix of both. The law remains the same regardless what the winter was or is like. Countries that have exceptionally warm winters do not really have a winter conditions so I'd guess their law wouldn't have a requirement for winter tires or chains

Yes of course, anyone who lives somewhere with winter knows that some winters are colder than others. In the late fall when it is time to swap the tires, you don’t know in advance whether the winter will be harsh or mild, but people put snow tires on anyway in places where harsh winters are common.

It would only make sense to require them every winter if the large majority of winters were quite cold. As such, nowhere in the US requires winter tires, but someone else noted that Quebec does. This makes sense, since even a mild Quebec winter will spend most of the time getting below freezing.

I would be very surprised if there were a place that required winter tires by law where the winter does not always get and stay at or below freezing for significant periods of time almost every winter.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#44

Post by sgerenser » Mon, Sep 14, 2026, 5:21 PM UTC

rjsw wrote: post #29

IsTom wrote: post #17

Earlier quoted context omitted.

> but I guess you get into problems with it being ice and snow that you're testing If it's not actively snowing for a few days roads get clear by thawing during day because of salt/cars being hot/sun shining. The "summer tyres get too hard in the cold" idea could very well come from places where the typical winter is at least < -5°C. It's not uncommon in central/eastern/northern europe for temperatures to be < -10°C…

The crossover point where winter tyres work better is < +7°C, it doesn't need to be below freezing.

Did you read the article? I thought his whole point is that little piece of folk wisdom didn't turn out to be true in actual testing.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#45

Post by RoddaWallPro » Mon, Sep 14, 2026, 5:23 PM UTC

mplanchard wrote: post #34

menaerus wrote: post #33

Earlier quoted context omitted.

If warranted by the law then yes, you have to abide to it.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

There are mountain passes in Oregon with posted signs during winter that say "winter tires or chains required". I'm not sure if you can actually be cited by an officer for not having them, but certainly you are a hazard to everyone on the road if you don't have them :)