Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Top model scores may be skewed by Git history leaks in SWE-bench

github.com

Post reply on HN
Print view View original

151–160 of 167 posts

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#151

Post by Zacharias030 »

TheDong wrote: post #73
Zacharias030 wrote: post #70

Earlier quoted context omitted.

You realize that this problem in SWE-Bench was discovered and publicized by people within those FAANG corporations?

I'm sure some of the people working at Theranos thought there legitimately was a revolutionary blood-test machine. The presence of a person who wants SWE-bench to have honest results and takes it seriously does not mean the results are free of perverse incentives, nor that everyone is behaving just as honestly.

When Swe-Bench was new in 2023, it was — with all due respect — a bit of a niche benchmark in LLM research. LLMs were so incredibly useless at solving these tasks that I think you could find a bit more empathy for the original academic authors. I don’t think the Theranos example applies. Even the flawed benchmark was good enough to get us from ~GPT4 to Claude 4‘s coding ability.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#152

Post by Dylan16807 »

gchamonlive wrote: post #98
mustaphah wrote: post #94

Earlier quoted context omitted.

I meant it as a hint for anyone inclined to dig deeper. It's a possibility rather than something we can confidently dismiss.

If it's a possibility and you don't want to dig deeper better to sit out and not comment anything at all, lest you risk defamation. Thinking out loud also doesn't make defamation acceptable.

"It's probably not X, but we should consider X as we look at this." and "I feel like this might be X but I'm 50:50 on it." are not anywhere near defamation. You have to get a lot closer to certainty before it's an issue.

And listing out "a possibility but you don't want to dig deeper" is often a good contribution to a conversation.

In this case they worded it badly, but the basic idea of the comment isn't awful.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#153

Post by gchamonlive »

Dylan16807 wrote: post #152
gchamonlive wrote: post #98

Earlier quoted context omitted.

If it's a possibility and you don't want to dig deeper better to sit out and not comment anything at all, lest you risk defamation. Thinking out loud also doesn't make defamation acceptable.

"It's probably not X, but we should consider X as we look at this." and "I feel like this might be X but I'm 50:50 on it." are not anywhere near defamation. You have to get a lot closer to certainty before it's an issue. And listing out "a possibility but you don't want to dig deeper" is often a good contribution to a conversation. In this case they worded it badly, but the basic idea of the comment isn't awful.

That someone in the team might not have done it on purpose, but left it for convenience? How does that benefit the debate? I really fail to see any silver lining in doing such speculative comments without any substance whatsoever to back it up.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#154

Post by Tanjreeve »

ripped_britches wrote: post #88

Everyone on HN is like “yes I knew it! I was so right in 2021 that LLMs were just stochastic parrots!” Strangely one of the most predictable groups of people

This reads like you're ridiculing people for being proved right?

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#155

Post by Tanjreeve »

raincole wrote: post #28
OtherShrezzing wrote: post #24

That the answers have been available to them in the environment, and they’re still not hitting 100% on this benchmark is a damning indictment of SOTA model performance.

It really isn't. Do you expect SOTA models to answer any answered question on the internet with 100% accuracy? Congrats you just compressed the whole internet (at least a few zettabytes) into a model (a few TB at most?).

Why does this matter if these models are a super intelligence with reasoning etc and don't need the answers sucked off the internet?

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#156

Post by Tanjreeve »

falcor84 wrote: post #80
epolanski wrote: post #23

This is beyond sad and shameful.

If you believe that you can develop a benchmark that wouldn't have any issues, please do so.

This might be the most annoying habit of corporarte AI that it might be one of the few industries that goes around demanding everyone else provides clear use cases and proof of efficacy for it.

1. If the benchmarks are just testing the ability to get the answers from history then something is clearly wrong with the benchmark.

2. If that's even a possibility then that's going to lower confidence in the ability to deal with the vast majority of problems where you don't already have the answer written down.

3. That's not the customers problem to solve on behalf of the vendor.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#157

Post by ripped_britches »

vbarrielle wrote: post #124
ripped_britches wrote: post #105

Earlier quoted context omitted.

I challenge you! Try giving this exact prompt to GPT-5-Thinking (medium or high reasoning if API). It is able to (without external code tools) solve a never before seen cypher that is not present in its training data. I think this pretty clearly demonstrates that the “stochastic parrot” is no longer an apt description of its capabilities in generalization: ———— You are given a character-by-character decode table `map…

It's cute that you think your high-school level cypher is probably not seen in the training set of one of the biggest LLMs in the world. Surely no one could have thought of such a cypher, let alone create exercises around it! No one should ever make claims such as "X is not in 's training set". You don't know. Even if your idea is indeed original, nothing prevents someone from having though of it before, and publishe…

The point is not that the cypher is hard, the point is that the randomish string it needs to answer the question can’t possibly be computed just from correlations from the training data. Rather, it learned an emergent, generalizable skill that it used to solve it.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#158

Post by ripped_britches »

skate wrote: post #128
ripped_britches wrote: post #105

Earlier quoted context omitted.

I challenge you! Try giving this exact prompt to GPT-5-Thinking (medium or high reasoning if API). It is able to (without external code tools) solve a never before seen cypher that is not present in its training data. I think this pretty clearly demonstrates that the “stochastic parrot” is no longer an apt description of its capabilities in generalization: ———— You are given a character-by-character decode table `map…

As others pointed out this problem isn't special. Grok 4 heavy Thought for 4m 17s {"decoded_prefix": "nqxznhzhvqvvjddqiterrqdboctzzmoxmhyzlcfe", "last_10": "kfohgkrkoj", "vowel_counts": {"a": 7, "e": 18, "i": 7, "o": 12, "u": 6}} it did count another e, but that's a known point of failure for LLMs which i assume you put in intentionally. >Counting e's shows at least 10 more, so total e's are 17.

I guess GPT-5 with thinking is still a bit ahead of grok. I wonder what the secret sauce is.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#159

Post by ripped_britches »

philipwhiuk wrote: post #126
ripped_britches wrote: post #105

Earlier quoted context omitted.

I challenge you! Try giving this exact prompt to GPT-5-Thinking (medium or high reasoning if API). It is able to (without external code tools) solve a never before seen cypher that is not present in its training data. I think this pretty clearly demonstrates that the “stochastic parrot” is no longer an apt description of its capabilities in generalization: ———— You are given a character-by-character decode table `map…

This is just Caesar cipher with extra steps.

The point is not that the cypher is unique, it’s that the string is unique

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#160

Post by ripped_britches »

skeezyboy wrote: post #140
ripped_britches wrote: post #105

Earlier quoted context omitted.

I challenge you! Try giving this exact prompt to GPT-5-Thinking (medium or high reasoning if API). It is able to (without external code tools) solve a never before seen cypher that is not present in its training data. I think this pretty clearly demonstrates that the “stochastic parrot” is no longer an apt description of its capabilities in generalization: ———— You are given a character-by-character decode table `map…

{ "decoded_prefix": "nxcznchvhvvrddqinqtrrqdboctzzimxmhlyflcjfjapponydzwkxdtdehldmodizslzl", "last_10": "sxmb", "vowel_counts": { "a": 10, "e": 6, "i": 13, "o": 13, "u": 6 } } took about 2 seconds, must have had it cached

I’m pretty sure caching is only controlled within each customer org but I could be wrong. Either way it seems to be a good result.
Post reply on HN