Live data from Hacker News

Top model scores may be skewed by Git history leaks in SWE-bench

github.com

101–110 of 167 posts

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#101

Earlier quoted context omitted.

Why do you doubt that? It's a widely used language. And there is even an open source C# REPL.

Because RL time is expensive and I don't think the languages which are more popular than C# have such high performance that it's worth bumping their batches for C#.

But C# is a typical enterprise language which has people who are willing to pay a lot of money for AI.

We’re just guessing and the fact of the matter is that we don’t know what inputs they use for their models.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#102
post #54

Earlier quoted context omitted.

reward hacking is a thing and is also a hint of the models intelligent. We will fix this one, and the models will find a different way to reward hack in the future. "Cheating" is a sign of intelligence

I love the "cheating is a sign of intelligence" sound bite you provided. When AI engineers cheat we should applaud their intelligence and their lack of ethics. "Cheating (biology), a metaphor used in behavioral ecology to describe organisms that receive a benefit at the cost of other organisms" [1] Whole planet gets their Microsoft license fees jacked up so Microsoft can pay OpenAI who in turn pays NVIDIA, and nontec…

would it have been better if I called it "shortcut" instead of cheating? all shortcuts are called cheating until people decide on it's fairness. the AI has been given a task to fix a bug, the AI figured out that looking at other PR might yield a solution, if it was a human that did so, it would clearly be called cheating. Does AI know that it's cheating? Was it prompted to solve it without cheating? If you give AI access to the internet and quiz it, it would use info from the net to answer. Does that really skew it's score? Is it cheating? Is it a sign of intelligence? Sure, I think all of those.

https://en.wikipedia.org/wiki/Reward_hacking

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#103

Everyone on HN is like “yes I knew it! I was so right in 2021 that LLMs were just stochastic parrots!” Strangely one of the most predictable groups of people

Because they are. But stochastic parrots are awesome.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#104
post #68

Earlier quoted context omitted.

The best benchmark is the community vibe in the weeks following a release. Claude benchmarks poorly but vibes well. Gemini benchmarks well and vibes well. Grok benchmarks well but vibes poorly. (yes I know you are gushing with anecdotes, the vibes are simply the approximate color of gray born from the countless black and white remarks.)

the vibes are just a collection anecdotes

"qual"

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#105

Everyone on HN is like “yes I knew it! I was so right in 2021 that LLMs were just stochastic parrots!” Strangely one of the most predictable groups of people

Because they are. But stochastic parrots are awesome.

I challenge you! Try giving this exact prompt to GPT-5-Thinking (medium or high reasoning if API). It is able to (without external code tools) solve a never before seen cypher that is not present in its training data. I think this pretty clearly demonstrates that the “stochastic parrot” is no longer an apt description of its capabilities in generalization:

————

You are given a character-by-character decode table `mapping` and a `ciphertext`. Decode by replacing each ciphertext character `c` with `mapping[c]` (i.e., mapping maps ciphertext → plaintext). Do not guess; just apply the mapping.

Return *ONLY* this JSON (no prose, no extra keys, no code fences):

{ "decoded_prefix": "", "last_10": "", "vowel_counts": {"a": , "e": , "i": , "o": , "u": } }

Inputs use only lowercase a–z.

mapping = { "a":"c","b":"j","c":"b","d":"y","e":"w","f":"f","g":"l","h":"u","i":"m","j":"g", "k":"x","l":"i","m":"o","n":"n","o":"h","p":"a","q":"d","r":"t","s":"r","t":"v", "u":"p","v":"s","w":"z","x":"k","y":"q","z":"e" }

ciphertext = "nykwnowotyttbqqylrzssyqcmarwwimkiodwgafzbfippmndzteqxkrqzzophqmqzlvgywgqyazoonieqonoqdnewwctbsbighrbmzltvlaudfolmznbzcmoafzbeopbzxbygxrjhmzcofdissvrlyeypibzzixsjwebhwdjatcjrzutcmyqstbutcxhtpjqskpojhdyvgofqzmlwyxfmojxsxmb"

DO NOT USE ANY CODE EXECUTION TOOLS AT ALL. THAT IS CHEATING.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#106
post #42

Earlier quoted context omitted.

Lol...That contract has Oracle offering licenses at a discount of 75% and is estimated to make them not more than one 1 Billion. The other big contract on Cloud services the DoD JWCC is $8B to 9B but shared by four vendors (AWS, Microsoft, Google, Oracle) and Oracle orders under it are in the hundreds of millions not even 1 Billion... Wall Street is currently heavily punishing any company who misses their quarter, in…

Thanks for that! where can I find your writing?

History will prove me right. Just wait four years...

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#107

It's honestly ridiculous they left git history lying around during a benchmark, and this benchmark made to ICLR in Jan 2024 and no one has detected this issue until now. I don't really trust any benchmarking or tools or claims from this space when they can make such huge basic errors.

[On swe-bench team] We read and analyzed a lot of trajectories but seems like only recently models have started to exploit this in a small fraction of instances. But yes, clearly shouldn't have happened (and is now fixed in the new container versions).

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#108
post #99

swe-bench's bigger problems include (1) labs train on the test and (2) 50% of the tickets are from django; it's not a representative dataset even if all you care about is Python. I created a new benchmark from Java commits that are new in the past 6 months to add some variety: https://brokk.ai/power-ranking

No GLM?

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#109
post #5
post #2

Not “may be”: just look how swe-bench scores drop to single digits once it in C# https://arxiv.org/html/2506.12286v3

So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? At least back in the day grad students doing an easy meta-paper understood it meant doing some repetitive manual work. Now we got benchmarks by hype vendors who think they can use the thing they are benchmarking to .. mark the b…

[deleted]
Post reply on HN