what is the difference of running the same code 5 times in parallel or running the same code 5 times sequentially?
Can LLMs write better code if you keep asking them to “write better code”?
161–170 of 461 posts
Re: Can LLMs write better code if you keep asking them to “write better code”?
#162This is an interesting read and it’s close to my experience that a simpler prompt with less or no details but with relevant context works well most of the time. More recently, I’ve flipped the process upside down by starting with a brief specfile, that is markdown file, with context, goal and usage example I.e how the api or CLI should be used in the end. See this post for details:
https://neoexogenesis.com/posts/rust-windsurf-transformation...
In terms of optimizing code, I’m not sure if there is a silver bullet. I mean when I optimize Rust code with Windsurf & Claude, it takes multiple benchmark runs and at least a few regressions if you were to leave Claude on its own. However, if you have a good hunch and write it as an idea to explore, Claude usually nails it given the idea wasn’t too crazy. That said, more iterations usually lead to faster and better code although there is no substitute to guiding the LLM. At least not yet.
Re: Can LLMs write better code if you keep asking them to “write better code”?
#163This kind of works on people too. You’ll need to be more polite, but asking someone to write some code, then asking if they can do it better, will often result in a better second attempt.
In any case, this isn’t surprising when you consider an LLM as an incomprehensibly sophisticated pattern matcher. It has a massive variety of code in its training data and it’s going to pull from that. What kind of code is the most common in that training data? Surely it’s mediocre code, since that’s by far the most common in the world. This massive “produce output like my training data” system is naturally going to tend towards producing that even if it can do better. It’s not human, it has no “produce the best possible result” drive. Then when you ask for something better, that pushes the output space to something with better results.
Re: Can LLMs write better code if you keep asking them to “write better code”?
#164> how to completely uninstall and reinstall postgresql on a debian distribution without losing the data in the database. https://www.phind.com/search?cache=lrcs0vmo0wte5x6igp5i3607 Still seem to struggle on basic instructions, and even understanding what it itself is doing. sudo rm -rf /etc/postgresql sudo rm -rf /var/lib/postgresql sudo rm -rf /var/log/postgresql > This process removes all PostgreSQL components, cle…
Is the problem that the antonym is a substring within "without losing the data in the database"? I've seen problems with opposites for LLMs before. If you specify "retaining the data" or "keeping the data" does it get it right?
Re: Can LLMs write better code if you keep asking them to “write better code”?
#1652 lessons to learn from this blog:
> these LLMs won’t replace software engineers anytime soon, because it requires a strong engineering background to recognize what is actually a good idea, along with other constraints that are domain specific.
> One issue with my experiments is that I’m benchmarking code improvement using Python, which isn’t the coding language developers consider when hyperoptimizing performance.
Re: Can LLMs write better code if you keep asking them to “write better code”?
#166Am I misinterpreting the prompt, or did the LLM misinterpret it from the get-go? Given a list of 1 million random integers between 1 and 100,000, find the difference between the smallest and the largest numbers whose digits sum up to 30. That doesn't read to me as "generate a list of 1 million random integers, then find the difference ..." but rather, "write a function that takes a list of integers as input". That sa…
I like the idea of your optimization, but it will not work as stated. The largest would be something close to MAXINT, the smallest 3999. With a range of 2 billion over 32 bits, the odds of both these being within a list of a million is quite a bit poorer than 99.9%.
The stated inputs are integers between 1 and 100,000, so if you're generating 1 million inputs, then you have 0.99999 ^ 1e6 = 4.5e-5 chance (roughly e^-10) of missing any given number, or roughly double that for missing any pair of values.
The key observation here is that you're sampling a relatively small space with a much greater number of samples, such that you have very high probability of hitting upon any point in the space.
Of course, it wouldn't work if you considered the full 32-bit integer space without increasing the number of samples to compensate. And, you'd need to be a little more clever to compute the largest possible value in your range.
Re: Can LLMs write better code if you keep asking them to “write better code”?
#167Something major missing from the LLM toolkit at the moment is that it can't actually run (and e.g. test or benchmark) its own code. Without that, the LLM is flying blind. I guess there are big security risks involved in making this happen. I wonder if anyone has figured out what kind of sandbox could safely be handed to a LLM.
I have experimented with using LLM for improving unit test coverage of a project. If you provide the model with test execution results and updated test coverage information, which can be automated, the LLM can indeed fix bugs and add improvements to tests that it created. I found it has high success rate at creating working unit tests with good coverage. I just used Docker for isolating the LLM-generated code from th…
Suggestion: Now take the code away, and have the chatbot generate code that passes the tests it wrote.
(In theory, you get a clean-room implementation of the original code. If you do this please ping me because I'd love to see the results.)
Re: Can LLMs write better code if you keep asking them to “write better code”?
#168I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…
[deleted]
Re: Can LLMs write better code if you keep asking them to “write better code”?
#169Re: Can LLMs write better code if you keep asking them to “write better code”?
#170Earlier quoted context omitted.
Even as someone with plenty of experience, this can still be a problem: I use them for stuff outside my domain, but where I can still debug the results. In my case, this means I use it for python and web frontend, where my professional experience has been iOS since 2010. ChatGPT has, for several generations, generally made stuff that works, but the libraries it gives me are often not the most appropriate, and are som…
I wish people would understand what a large language model is. There is no thinking. No comprehension. No decisions. Instead, think of your queries as super human friendly SQL. The database? Massive amounts of data boiled down to unique entries with probabilities. This is a simplistic, but accurate way to think of LLMs. So how much code is on the web for a particular problem solve? 10k blog entries, stackoverflow res…
> I wish people would understand what a large language model is.
I think your view of llm does not explain the learning of algorithms that these constructs are clearly capable of, see for example: https://arxiv.org/abs/2208.01066
More generally, the best way to compress information from too many different coding examples is to figure out how to code rather than try to interpolate between existing blogs and QA forums.
My own speculation is that with additional effort during training (RL or active learning in the training loop) we will probably reach superhuman coding performance within two years. I think that o3 is still imperfect but not very far from that point.