Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If AI writes your code, why use Python?

medium.com

Post reply on HN
Print view View original

961–970 of 1001 posts

Re: If AI writes your code, why use Python?

#961

Post by stingraycharles »

shimman wrote: post #919
stingraycharles wrote: post #369

Earlier quoted context omitted.

I’m super surprised that C++ scores so high, this does not match our experience at all, and for anything performance critical it always drops the ball completely. I also don’t understand how these “games” map to real world complex problems. How are you measuring success? How does “adversarial customer service” map to “this LLM is better at C++ than the other” ? How are you sure you’re not just benchmarking language s…

It's a VC backed company using "scoreboards" as a means of marketing. There is nothing scientific or academic about this, stop assuming as much.

You’re probably correct, but the problem I have is that even a lot of “scientific” benchmarks have similar issues. The whole AI benchmark landscape seems to suffer from similar issues, that benchmarks don’t appear to be matching real world experience (eg open weight Chinese models scoring super high on all benchmarks, while clearly being inferior when using in real life).

Re: If AI writes your code, why use Python?

#962

Post by gertlabs »

stingraycharles wrote: post #959
gertlabs wrote: post #835

Earlier quoted context omitted.

I'm not so sure there's a difference. The main thing we want to measure for LLMs is broad reasoning capability, but seeing how that ability changes under different constraints (like programming language) is the interesting part.

Ok, when you put it that way, I agree that this is in fact interesting. Maybe you should actually put that sentence in some kind of form on your website because it clarifies a lot.

Appreciate the feedback, we're taking a lot of good recommendations from this thread, which will be live in a few days: better explanations/tooltips, more samples for any filter sets we offer (or show sample counts if potentially noisy), and adding some functional languages like Clojure

Re: If AI writes your code, why use Python?

#963

Post by antonvs »

doug_durham wrote: post #909
gertlabs wrote: post #174

Earlier quoted context omitted.

Surprisingly, LLMs are actually much worse at reasoning in Python than other common programming languages for agentic coding tasks. Data here: https://gertlabs.com/rankings?mode=agentic_coding

Reasoning??? Since when to LLMs reason. The word "reasoning" has become almost worthless in modern computer science due to misuse.

LLMs can regularly produce correct conclusions from the input they’re given. When a human does that, we call the process involved “reasoning”. LLMs can even explain how their conclusions are supported in ways that again, if a human employed them, would be called reasoning.

As such, functionally speaking - focusing on inputs and outputs - models can be said to reason. If you’re tempted to argue that it’s “only statistics” or similar, consider that human reasoning is only chemical and electrical signals in neurons.

Re: If AI writes your code, why use Python?

#964

Post by antonvs »

by364 wrote: post #399
js8 wrote: post #301

Earlier quoted context omitted.

The LLMs are generally still pretty bad at (deductive) reasoning. IME they go along more with the things like variable names and comments than the actual program logic (it would be an interesting experiment to compare LLM's understanding of three identical programs with different identifiers, one with normal identifiers, one with obfuscated identifiers, and one with deliberately misleading identifiers). I also think…

wait till you look inside a neural network and realize they're incapable of deductive reasoning! amazing how many devs that talk about "AI" would probably have a hard time telling apart deductive and inductive reasoning.

> wait till you look inside a neural network and realize they're incapable of deductive reasoning!

You could just as well say, wait until you look inside a human brain and realize they’re incapable of deductive reasoning!

Of course we’re not actually able to do that, but if you could you might very well find a similar apparent lack of whatever it is you expect to see at that level.

Re: If AI writes your code, why use Python?

#967

Post by jmalicki »

antonvs wrote: post #940
echelon wrote: post #646

Earlier quoted context omitted.

You'll learn Rust faster with AI and should be able to solve it yourself. You're unlikely to wind up in such a situation though. The design work Claude does in Rust is really sensible and idiomatic, and I really don't think you'll be unable to refactor or redesign things. Claude is extremely good with Rust generation, refactoring, and manipulation. I'll go as far as to say that AI has removed most of the complaints p…

> AI has removed most of the complaints people had with learning or using Rust. Not even close. The problem is not only writing the code, it’s being able to review and understand it. An AI can’t magically transfer that knowledge into someone’s brain, they have to develop that knowledge themselves. I’m dealing with this currently with a team of ML developers. A couple of them like Rust, the rest don’t know it. They’ve…

That's not lack of knowledge of a specific language, that's just not knowing how to code period.

Anything in any language where it is trying to change 40 files for something simple should have you stopping.

Re: If AI writes your code, why use Python?

#968

Post by kangaroux »

timbaboon wrote: post #907
t43562 wrote: post #655

If AI writes your code, why use Rust? Why not use assembler? Why waste time trolling people that your one true language is the answer for LLMs when your view of the future is: no more programming full stop.

I gave exactly the same comment to a colleague at work today :D

[dead]

Re: If AI writes your code, why use Python?

#969

Post by dnautics »

nojito wrote: post #954
dnautics wrote: post #274

Earlier quoted context omitted.

and yet dynamically typed elixir wipes the floor with go. https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/blob/ma...

This doesn't say what you think it does. Language like elixir were given easier problems to solve when compared to other languages.

Where does the paper say this?

Re: If AI writes your code, why use Python?

#970

Post by rightbyte »

antonvs wrote: post #963
doug_durham wrote: post #909

Earlier quoted context omitted.

Reasoning??? Since when to LLMs reason. The word "reasoning" has become almost worthless in modern computer science due to misuse.

LLMs can regularly produce correct conclusions from the input they’re given. When a human does that, we call the process involved “reasoning”. LLMs can even explain how their conclusions are supported in ways that again, if a human employed them, would be called reasoning. As such, functionally speaking - focusing on inputs and outputs - models can be said to reason. If you’re tempted to argue that it’s “only statist…

I think I've read some study that the LLM reasoning is a mirage.
Post reply on HN