SWE-bench Verified no longer measures frontier coding capabilities
1–10 of 209 posts
Re: SWE-bench Verified no longer measures frontier coding capabilities
#2I don't understand these websites which force translation to my native language.
I mean, it's fine as it's useful for many people, but where is the button for disabling it ? Or why is it enabled by default ?
"codage de pointe" sounds so weird and cringe in French.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#3This feels very much like "we are now moving the goal posts".
Re: SWE-bench Verified no longer measures frontier coding capabilities
#4This feels very much like "we are now moving the goal posts".
But this is the good kind of goalpost moving
Re: SWE-bench Verified no longer measures frontier coding capabilities
#5Terminal Bench is the future
Re: SWE-bench Verified no longer measures frontier coding capabilities
#6I don't understand these websites which force translation to my native language. I mean, it's fine as it's useful for many people, but where is the button for disabling it ? Or why is it enabled by default ? "codage de pointe" sounds so weird and cringe in French.
Same for apps and games. I understand English just fine, no need to switch to your shitty Google-translate localization just because my iPhone or PlayStation is set to my native language.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#7I don't understand these websites which force translation to my native language. I mean, it's fine as it's useful for many people, but where is the button for disabling it ? Or why is it enabled by default ? "codage de pointe" sounds so weird and cringe in French.
Does your browser request French via an Accept-Language header perhaps? What really infuriates me is when sites don’t respect that header and give you a translation based on IP location.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#8I don't understand these websites which force translation to my native language. I mean, it's fine as it's useful for many people, but where is the button for disabling it ? Or why is it enabled by default ? "codage de pointe" sounds so weird and cringe in French.
Does your browser request French via an Accept-Language header perhaps? What really infuriates me is when sites don’t respect that header and give you a translation based on IP location.
Regardless if it does or not, users should be able to manually override what language the website is in, at least be able to read the native one, regardless of what the original language was, what headers you send and where geodatabases think your IP is from.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#9Terminal Bench is the future
First, you might want to say why you think so, otherwise this is just borderline spam. Secondly, when your praise things (without motivation or reasoning even), and you've contributed to that specific thing, please say that up front instead of just praising the thing, again it makes it look like spam otherwise.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#10This feels very much like "we are now moving the goal posts".
But this is the good kind of goalpost moving
Only if you didn't read the article.
They're saying they need to move on from it because the benchmark is flawed (without bringing in proof) and that's why they can't hit 100%.
It's not a "our models are so good that the benchmark is too easy" thing.