Konwinski Prize
andykonwinski.com
Konwinski Prize
1–10 of 49 posts
Re: Konwinski Prize
#2Fabulous. A big round of applause for Andy Kowinski and the SWE-bench folks!
Re: Konwinski Prize
#3The Kaggle competition page has more details: https://www.kaggle.com/competitions/konwinski-prize
The prizes scale with the model’s score; the total prize pool is between $100,000 and $1,225,000, depending on the top scores.
Re: Konwinski Prize
#4Surprised to see Amazon Q Developer already at 55% on the verified suite.
But what I appreciate even more is that we keep pushing the bar for what an AI can/should be able to do. Excited to track this benchmark over time.
Re: Konwinski Prize
#5The Kaggle competition page has more details: https://www.kaggle.com/competitions/konwinski-prize The prizes scale with the model’s score; the total prize pool is between $100,000 and $1,225,000, depending on the top scores.
>$1M for the AI that can close 90% of new GitHub issues
If your AI can do this, it's worth several orders of magnitude more. Just FYI.
Re: Konwinski Prize
#6Very cool to see "outcome oriented" prizes like this -- it's another way to fund research, perhaps. Will be curious to track who does this and whether success in the prize correlates with deep innovation ...
Re: Konwinski Prize
#7SWE-bench with a private final eval, so you can't hack the test set!
In a perfect world this wouldn't be necessary, but in the current research environment where benchmarks are the primary currency and are usually taken at face value, more unbiased evals with known methodology but hidden tests are exactly what we need.
Also one reason why, for instance, I trust small but well-curated benchmarks such as Aider (https://aider.chat/docs/leaderboards/) or Wolfram (https://www.wolfram.com/llm-benchmarking-project/index.php.e...) over large, widely targeted, and increasingly saturated or gamed benchmarks such as LMSYS Arena or HumanEval.
Goodhart's law is thriving and it's our duty to fight it.
Re: Konwinski Prize
#8The Kaggle competition page has more details: https://www.kaggle.com/competitions/konwinski-prize The prizes scale with the model’s score; the total prize pool is between $100,000 and $1,225,000, depending on the top scores.
>$1M for the AI that can close 90% of new GitHub issues If your AI can do this, it's worth several orders of magnitude more. Just FYI.
Exactly, I'll personally buy it for $2Million for anyone that can get it and assign me the full code/weight and rights.
Re: Konwinski Prize
#9What would be an example of cheating since it says "no cheating".
Re: Konwinski Prize
#10What would be an example of cheating since it says "no cheating".
The only reasonable way to cheat on this would be to find real bugs in many repos, train your models on the solutions, wait till the cut-off period, report those bugs, propose PRs and hope your bugs get selected. Pretty small chances, tbh and probably not worth the rewards (the 90% solve rate is pretty much impossible given the constraints - 4x L4s and ~4-6min / problem. There's no way any models that can be ran on those machines under those time limits are that much better than the SotA frontier models)