Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires • Hacker News Board

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#41

Post by pm215 » Mon, Sep 14, 2026, 3:41 PM UTC

rjsw wrote: post #29

IsTom wrote: post #17

Earlier quoted context omitted.

> but I guess you get into problems with it being ice and snow that you're testing If it's not actively snowing for a few days roads get clear by thawing during day because of salt/cars being hot/sun shining. The "summer tyres get too hard in the cold" idea could very well come from places where the typical winter is at least < -5°C. It's not uncommon in central/eastern/northern europe for temperatures to be < -10°C…

The crossover point where winter tyres work better is < +7°C, it doesn't need to be below freezing.

The linked article claims "in dry conditions, summer tires have the best grip down to 0C", though, which doesn't seem to match where you suggest the crossover point is.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#42

Post by mplanchard » Mon, Sep 14, 2026, 3:54 PM UTC

auxym wrote: post #37

mplanchard wrote: post #34

Earlier quoted context omitted.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

Quebec, Canada mandates winter tires between December 1st and March 15th. (But the winters are cold).

Didn’t know it was required, but it makes sense, as it is indeed quite cold. Good thing I always have winter tires on in the winter anyway, so I’m not breaking the law when I drive to Montreal!

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#43

Post by mplanchard » Mon, Sep 14, 2026, 3:59 PM UTC

menaerus wrote: post #36

mplanchard wrote: post #34

Earlier quoted context omitted.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

Sometimes the winters may be cold, some other times they may be exceptionally warm, and sometimes they're a mix of both. The law remains the same regardless what the winter was or is like. Countries that have exceptionally warm winters do not really have a winter conditions so I'd guess their law wouldn't have a requirement for winter tires or chains

Yes of course, anyone who lives somewhere with winter knows that some winters are colder than others. In the late fall when it is time to swap the tires, you don’t know in advance whether the winter will be harsh or mild, but people put snow tires on anyway in places where harsh winters are common.

It would only make sense to require them every winter if the large majority of winters were quite cold. As such, nowhere in the US requires winter tires, but someone else noted that Quebec does. This makes sense, since even a mild Quebec winter will spend most of the time getting below freezing.

I would be very surprised if there were a place that required winter tires by law where the winter does not always get and stay at or below freezing for significant periods of time almost every winter.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#44

Post by sgerenser » Mon, Sep 14, 2026, 5:21 PM UTC

rjsw wrote: post #29

IsTom wrote: post #17

Earlier quoted context omitted.

> but I guess you get into problems with it being ice and snow that you're testing If it's not actively snowing for a few days roads get clear by thawing during day because of salt/cars being hot/sun shining. The "summer tyres get too hard in the cold" idea could very well come from places where the typical winter is at least < -5°C. It's not uncommon in central/eastern/northern europe for temperatures to be < -10°C…

The crossover point where winter tyres work better is < +7°C, it doesn't need to be below freezing.

Did you read the article? I thought his whole point is that little piece of folk wisdom didn't turn out to be true in actual testing.

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#45

Post by RoddaWallPro » Mon, Sep 14, 2026, 5:23 PM UTC

mplanchard wrote: post #34

menaerus wrote: post #33

Earlier quoted context omitted.

If warranted by the law then yes, you have to abide to it.

Are there any places that require winter tires by law that don’t have cold winters? I don’t know of anywhere in the US that even requires winter tires, although I think some states require you to have winter tires OR chains.

There are mountain passes in Oregon with posted signs during winter that say "winter tires or chains required". I'm not sure if you can actually be cited by an officer for not having them, but certainly you are a hazard to everyone on the road if you don't have them :)

Re: Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

#46

Post by cb321 » Mon, Sep 14, 2026, 5:39 PM UTC

lh712 wrote: post #40

cb321 wrote: post #24

Earlier quoted context omitted.

Yeah. My point was mostly to expand upon your original post - sorry if it sounded like I disagreed. You say the same in your original "iteration of speculation and observation". Everything else is about what "derive" and "knowledge" might mean (I agree "medieval" usually means pre-Renaissance and Galileo is modern-era), how much empirical "proof" is in "derive", etc. However, all you need for "possible" is "the idea"…

Thank you for your reply! Yes, I fully agree that meaning of "derive" and other related concepts are non-trivial. (And I actually changed my own "working definition" between my two posts; while in the first one I was leaning more towards theories anchored by observation, even if indirectly, in the second one I slipped into more mathematical/philosophical perspective. The second part of my first reply makes sense only…

You're welcome.

Another lesser known wrinkle along these lines is that if more of Aristarchus of Samos' work on heliocentricity had been developed into a "calculational framework" for planetary motion by a contemporaneous Ptolemy competitor (a la Copernicus in 1543), various relativity ideas might well have started in 270BC instead of 1600 AD.

Aristarchus was already WAY ahead of a few games - making a guess that "absolute rest" was illusory and tiny stars were similar to giant Sol. If not for Archimedes' serious star power, we might not even know of Aristarchus' heliocentric ideas! Relativity ideas of various kinds are short intellectual hops from "the absolute rest of your intuition is illusory".

Aristarchus and his supporters just had to assert The Stars were very distant -- far enough for there to be NO PARALLAX! Various "coincidences" all conspired to suppress Aristarchus' plausibility, like: A) Just HOW DISTANT stars are/local galactic stellar density &| B) poor human VISUAL ACUITY relative to C) Earth ORBIT vs. Sun LUMINOSITY (parallax baseline) & maybe existence of Moon to vent atmo &| D) VERY SLOW (3 millennium) development from near prehistoric glass (1500BC) to grinding lenses for human eyes in the 1300s AD leading to Galileo's telescopes & etc. If ANY of (A)..(D) were about ~10x better (ALL of which are imaginably so), the day that universe changed could've been much earlier. Heck, even if Aristarchus just had a really good spin doctor like a major religion pushing the plausibility of "Sun=A close Star", that might've been enough.

Archimedes almost invented the underlying ideas of integration and limits and all that, too. Close but no cigar, but (had that been in hand) analytic geometry and differential equations are short steps away. I mean, Newton surely noted the equivalence of gravitational and inertial "mass" (charge vs. kinematics) which is the weak principle of equivalence of GR, after all. Like I said - "littered". ;-) There are probably whole books written on the topic (or adjacent topics) of "All the Things Humanity Nearly Figured Out Earlier" that have a more historical bent than the usual "sci-fi tilt".