[dead]
SWE-bench Verified no longer measures frontier coding capabilities
181–190 of 209 posts
Re: SWE-bench Verified no longer measures frontier coding capabilities
#182[dead]
Re: SWE-bench Verified no longer measures frontier coding capabilities
#183[dead]
Re: SWE-bench Verified no longer measures frontier coding capabilities
#184Earlier quoted context omitted.
What do you mean with this? What is this world model, what does it capture?
You keep a document going called "state of the world", on every turn, you read this document in (as context), use it to help compute what happens, and based on what happens, create an updated "state of the world" document. You track important details so your LLM is consistent from turn to turn. If you doing an RPG, which I guess is where this is more obvious, you track the play and enemy positions, their health, thei…
I would love to see consistent-world-state-capturing more integrated into, for example, SillyTavern.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#185Earlier quoted context omitted.
This is why I made Zork bench. Zork, the text adventure game, is in the training data for LLMs. It’s also deterministic. Therefore it should be easy for an LLM to play and complete. Yet they don’t. Understanding why is the goal of Zork bench. https://github.com/mnky9800n/zork-bench
Actually the Zorks weren't deterministic, especially Zork II. The Wizard could F you over pretty badly if he appeared at an inopportune time.
Was that using an RNG? Or is the entire game deterministic?
Re: SWE-bench Verified no longer measures frontier coding capabilities
#186> We audited a 27.6% subset of the dataset that models often failed to solve and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions, despite our best efforts in improving on this in the initial creation of SWE-bench Verified. Is this saying a quarter* of the questions and answers were wrong, this whole time?! If so, how was this ever, in any way, a va…
> Is this saying a quarter of the questions and answers were wrong, this whole time?! No, they're saying 59.4% of the 27.6% subset had flawed test cases I think. > If so, how was this ever, in any way, a valid measurement? Benchmarks essentially aren't, for practical concerns anyways. They don't represent your use case, and they don't represent any and all use cases, they're valid for measuring exactly what's include…
the ecosystem obsession with public benchmarks comes from the fact that running benchmark costs, and labs don't test on any given private benchmark
but yeah you're correct anyone optimizing for public-bench rank instead of their own task-distribution eval has been pointing at the wrong thing for a while
still I guess useful signal to know which one model to consider, negative signal is still signal, assuming everyone is gaming benchmark in certain ways, lack of performance do result in a real workload effect
Re: SWE-bench Verified no longer measures frontier coding capabilities
#187Earlier quoted context omitted.
It’s just hard to make them not part of the training data. We see this a bit with BrowseComp plus and other deep research datasets. Not because frontier labs are trying to cheat, but just from training on the full web. You need new datasets perpetually.
Or hidden benchmarks, though it's then harder to get people to trust the results.
How do you hide them if you aren't self hosting the model?
Re: SWE-bench Verified no longer measures frontier coding capabilities
#188Earlier quoted context omitted.
I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for it.
i disagree: https://www.philosophicalhacker.com/post/anthropic-error/
I don't understand that methodology in the first place. Does Anthropic even have some kind of somewhat objective definition to measure and judge "memorization"? Is there any evidence that other LLMs are viable tool to determine that?
Re: SWE-bench Verified no longer measures frontier coding capabilities
#189[deleted]
Re: SWE-bench Verified no longer measures frontier coding capabilities
#190I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…
They're not saying "Don't use SWE-bench Verified because it's saturated".
They're saying:
1. A large number of the tests are inaccurate; so correct solutions will be marked as incorrect.
2. Frontier models have already read and memorized the PR's the problems are based on.
3. In fact, many problems are essentially impossible to get right if you haven't memorized the solution: for example, the test cases will fail if you didn't happen to expose a helper function with a specific name. That name isn't mentioned in the problem; but frontier models are passing that test anyway because they remember that such a helper function is necessary.
If the next stage of benchmarks don't address these issues, they'll continue to have the same problems, saturated or not.