251–260 of 461 posts
I've had great luck with Cursor by simply cursing at it when it makes repeated mistakes. I'll speak to it like a DI would speak to a recruit a basic training. And it works. I was speaking to some of the Cursor dev team on Discord, and they confirmed that being aggressive with the AI can lead to better results.
Am I misinterpreting the prompt, or did the LLM misinterpret it from the get-go? Given a list of 1 million random integers between 1 and 100,000, find the difference between the smallest and the largest numbers whose digits sum up to 30. That doesn't read to me as "generate a list of 1 million random integers, then find the difference ..." but rather, "write a function that takes a list of integers as input". That sa…
This was the intent and it's indeed a common assumption for a coding question job interviews, and notably it's fixed in the prompt-engineered version. I didn't mention it because it may be too much semantics as it doesn't affect the logic/performance, which was the intent of the benchmarking.
I've had great luck with Cursor by simply cursing at it when it makes repeated mistakes. I'll speak to it like a DI would speak to a recruit a basic training. And it works. I was speaking to some of the Cursor dev team on Discord, and they confirmed that being aggressive with the AI can lead to better results.
This makes me sad. Have you tried being really nice and supportive instead? I really don't want to have to yell at my computer for it to work :(
This seems like anthromorphizing the model ... Occam's Razor says that the improvement coming from iterative requests to improve the code comes from the incremental iteration, not incentivizing the model to do it's best. If the latter were the case then one could get the best version on first attempt by telling it your grandmother's life was on the line or whatever. Reasoning is known weakness of these models, so jum…
That said, it was done with ChatGPT 3.5/4, I suspect Claude 3.5 Sonnet would behave much different.
Earlier quoted context omitted.
Because otherwise we are talking about LLMs augmented with external tools (i.e. Python interpreters). My original comment was pointing to the limitations of LLMs in writing code by themselves.
You wouldn't ask a programmer to solve a problem and then also not let them write down the source or debug the program as you write it? Are you asking it to not write down an algorithm that is general? They are doing a pretty good job on mathematical proofs. I still don't understand why you wouldn't let its full reasoning abilities by letting it write down code or even another agent. We should be testing towards the…
It is worth noting that even ChatGPT-o1 doesn't seem capable of finding this code optimization, despite having access to a Python interpreter.
Earlier quoted context omitted.
Hmm, I worked with students in an “intro to programming” type course for a couple years. As far as I’m concerned, “I added complexity until it compiled and now it works but I don’t understand it” is pretty close to passing the Turing test, hahaha.
Oh yeah, it's definitely junior dev behaviour. The problem is that it's junior dev behaviour!
Earlier quoted context omitted.
Because you cite is about: > in-context learning LLMs have no concept of the symantic meaning of what they do, they just are dealing with next token prediction. "in-context learning" is the problem, not the solution to general programming tasks. Memoryless, ergodic, sub Turing complete problems are a very tiny class. Think about how the Entscheidungsproblem relates to halting or the frame problem and the specificatio…
Here is a fairly good lecture series on graduate level complexity theory that will help understand parts. At least why multiple iterations help but why they also aren't the answer to super human results. https://youtube.com/playlist?list=PLm3J0oaFux3b8Gg1DdaJOzYNs...
2 lessons to learn from this blog: > these LLMs won’t replace software engineers anytime soon, because it requires a strong engineering background to recognize what is actually a good idea, along with other constraints that are domain specific. > One issue with my experiments is that I’m benchmarking code improvement using Python, which isn’t the coding language developers consider when hyperoptimizing performance.
TBH I'm not sure how he arrived at "won’t replace software engineers anytime soon" The LLM solved his task. With his "improved prompt" the code is good. The LLM in his setup was not given a chance to actually debug its code. It only took him 5 "improve this code" commands to get to the final optimized result, which means the whole thing was solved (LLM execution time) in under 1 minute.
My comments on "what you are not sure" is that Max is a software engineer (I am sure a good one) and he kept iterating the code until it reached close to 100x faster code because he knew what "write better code" looked like.
Now ask yourself this question: Is there any chance a no-code/low-code developer will come to a conclusion deduced by Max (he is not the only one) that you are not sure about?
An experienced software engineer/developer is capable of improving LLM written code into better code with the help of LLM.
I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…
I often run into LLMs writing "beginner code" that uses the most fundamental findings in really impractical ways. Trained on too many tutorials I assume. Usually, specifying the packages to use and asking for something less convoluted works really well. Problem is, how would you know if you have never learned to code without an LLM?
That and unit tests. I write the first table based test case, then give it the source and the test code, and ask it to fill it in with more test cases.