41–50 of 60 posts
The best submission is swe-bench-multilingual is Claude 3.7 Sonnet which solves ~43% of the issues in the dataset.
Unlike most SWE bench submissions, Qodo Command one uses the product directly. I think that the next step is getting an official "checked" mark by the SWE bench team
I feel like the bash only SWE Bench Verified (a.k.a model + mini-swe-agent) is the closest thing to measuring the inherent ability of the model vs. the scaffolding. https://github.com/SWE-agent/mini-swe-agent
Earlier quoted context omitted.
This is classic Goodhart's law. "When a measure becomes a target, it ceases to be a good measure" https://en.wikipedia.org/wiki/Goodhart%27s_law
It's really not that hard to not build a custom bench setup to game the benchmark instead of just using your product straight out of the box, though.
Earlier quoted context omitted.
From my perspective as a potential user the number of attempts is the number of times I have to tell it what to do. If you have an agent that makes a single attempt and is 60% accurate vs another that makes 5 attempts and is 80% accurate, why would you care that each individual attempt of the 2nd model is less accurate than the first?
This ok from your perspective then? def make_pass@1_agent(agent, n): def retry_agent(problem): for attempt in range(n): result = agent(problem) if result.success: return result return result return retry_agent
Think of the agent like an employee. If he delivers the code within the expected time and to the expected quality standards, his process of getting there means almost nothing. Do I care if he tried 4 different approaches along the way and threw out the first 3? Not a bit.
I'm curious how do these LLM wrapper companies think they'll survive long term - especially coding related wrappers. I could understand focusing on a niche business use case, but coding is a main focus of the foundation models themselves.
I'm curious how do these LLM wrapper companies think they'll survive long term - especially coding related wrappers. I could understand focusing on a niche business use case, but coding is a main focus of the foundation models themselves.
Labeling them as “wrappers” and “niche business” indicates a strong cognitive bias already. Value can be created on both sides of the equation.
Earlier quoted context omitted.
Labeling them as “wrappers” and “niche business” indicates a strong cognitive bias already. Value can be created on both sides of the equation.
How so? They are wrappers, and it is niche.