Would love to see how DeepSeek R1 compares to O1 here.
ASTRA: HackerRank's coding benchmark for LLMs
11–16 of 16 posts
Re: ASTRA: HackerRank's coding benchmark for LLMs
#12> To mimic real-world development, HackerRank’s ASTRA Benchmark Dataset includes, on average, 12 source code and configuration files per question as model inputs.
How is 12 files "real-world development"? My hobby project currently has 142 files and most non-trivial changes would involve adding a new file. My small work project has 79 and similarly, any non-trivial changes will need to add a file. These are small codebases. My previous team was ~450k lines across ~thousands of files, and we managed that pretty effectively with 6 engineers.
Getting the right answer out of an LLM for these sorts of tasks is fine if you give it little enough context that it's an effectively greenfield task as most of these problems end up being. But giving them a whole codebase and expecting the right answer, or the process of choosing the right subset to give them, are still big unsolved problems.
At this point it honestly feels a bit like gaslighting, suggesting that a 12 file NodeJS server is representative of software engineering.
Re: ASTRA: HackerRank's coding benchmark for LLMs
#13> To mimic real-world development, HackerRank’s ASTRA Benchmark Dataset includes, on average, 12 source code and configuration files per question as model inputs. How is 12 files "real-world development"? My hobby project currently has 142 files and most non-trivial changes would involve adding a new file. My small work project has 79 and similarly, any non-trivial changes will need to add a file. These are small cod…
[deleted]
Re: ASTRA: HackerRank's coding benchmark for LLMs
#14How will programming change when we reach reach 99-100%?
Today I, in a fit of laziness, asked claude for a c function to invert a matrix instead of writing it myself. It gave me a function that is wrong if malloc gives a pointer to non-zeroed memory. If its 99% and the 1% continue to be mistakes like that, programming is going to be hell.
As long as you can analyze and fix those kind of issues, your skill should be in high demand given current developments.
Re: ASTRA: HackerRank's coding benchmark for LLMs
#15Seems like a very limited subset of software development to be basing a benchmark on Where’s the kernel dev? Where’s the embedded dev? Where’s the throwaway python script?
We all know that nobody actually writes kernels, compilers, firmware, or video games, they're just mined straight from the earth.
Re: ASTRA: HackerRank's coding benchmark for LLMs
#16How will programming change when we reach reach 99-100%?
Today I, in a fit of laziness, asked claude for a c function to invert a matrix instead of writing it myself. It gave me a function that is wrong if malloc gives a pointer to non-zeroed memory. If its 99% and the 1% continue to be mistakes like that, programming is going to be hell.
If Claude can spit that out , I am more than competent enough to review it. Saving 15-20m here and there is a good trade off IMO.