Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS

swe-rebench.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: 13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS

#11

Post by ducktective »

spullara wrote: post #2

They are all different problems for the different languages. I was hoping this was a benchmark that attempted to see which languages were more efficient to use with which models.

Exactly. I was wondering what language would be better for heavy agentic workflow. Due to a recent blog post, Clojure seem to be the most terse [1]. But some people are suggesting static types would help in this workflow.

[1] : https://martinalderson.com/posts/which-programming-languages...

Re: 13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS

#13

Post by revetkn »

ducktective wrote: post #11
spullara wrote: post #2

They are all different problems for the different languages. I was hoping this was a benchmark that attempted to see which languages were more efficient to use with which models.

Exactly. I was wondering what language would be better for heavy agentic workflow. Due to a recent blog post, Clojure seem to be the most terse [1]. But some people are suggesting static types would help in this workflow. [1] : https://martinalderson.com/posts/which-programming-languages...

I'm curious about this as well. I use Java quite a bit and I feel like my results are excellent, but colleagues who use other languages are less impressed. I wonder if this is because Java is more amenable to agentic workflows, or there is a large corpus of example code, or some other reason.

Re: 13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS

#15

Post by joshka »

goldenarm wrote: post #7

Why are models better than agents, isn't it supposed to be the opposite? I don't understand the difference and what you are measuring.

Some agents have specific tools that the models have been trained to use. E.g. diff formats for editing that aren't the same as the standard unified diff format. Access to specific thread / subagent / etc. tooling or the base prompt can perhaps also impact how the tasks are completed.

Re: 13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS

#16

Post by guywithahat »

Consistently impressed with the performance of Grok, given they started so much later than everyone else and, in some ways, are less well funded compared to OpenAI and Anthropic. I wonder how much of this is just luck, name recognition, or management style. Marc Andreesen likes to talk about how Elon companies have a unique engineering-heavy management structure, which contrasts the research heavy cultures of OpenAI and other labs, and it makes me wonder if that sort of thing could be behind their relative success.

Re: 13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS

#17

Post by pseudosavant »

guywithahat wrote: post #16

Consistently impressed with the performance of Grok, given they started so much later than everyone else and, in some ways, are less well funded compared to OpenAI and Anthropic. I wonder how much of this is just luck, name recognition, or management style. Marc Andreesen likes to talk about how Elon companies have a unique engineering-heavy management structure, which contrasts the research heavy cultures of OpenAI…

Distillation. It isn’t just Chinese models doing that to OpenAI and Anthropic.
Post reply on HN