11–20 of 33 posts
> it's not clear if GPT will have sufficient attention to handle the more complex cases.
I think this could be helped with some tooling, by giving it smaller pieces of code to digest, in a new prompt and a somewhat smaller prompt. These LLM can't seem to dive into cracks if the context isn't constrained to that crack.
One area of low hanging fruit here is to automatically evaluate the quality/accuracy/correctness/etc of a given generation and select (or merge) between multiple possibilities generated in parallel. Sometimes it will forget to fill in a function def, so use the one from iter #3, etc. You could go so far as to run candidates in a sandbox with an input and evaluate which one gives output -- ideally which output is closest to what is desired, if you can define that.
Also a sort of "whitelist" for valid functions and routines - sometimes it's close not still wrong, if you can map the hallucinations and mistakes to what it's supposed to be, that also can probably go a long way.
I don't see any iterations. I can sometimes get it to improve the code just by telling it that there's a problem, or to identify any problems. > it's not clear if GPT will have sufficient attention to handle the more complex cases. I think this could be helped with some tooling, by giving it smaller pieces of code to digest, in a new prompt and a somewhat smaller prompt. These LLM can't seem to dive into cracks if th…
how will you identify that there's a problem?
I don't see any iterations. I can sometimes get it to improve the code just by telling it that there's a problem, or to identify any problems. > it's not clear if GPT will have sufficient attention to handle the more complex cases. I think this could be helped with some tooling, by giving it smaller pieces of code to digest, in a new prompt and a somewhat smaller prompt. These LLM can't seem to dive into cracks if th…
> just by telling it that there's a problem how will you identify that there's a problem?
I don't see any iterations. I can sometimes get it to improve the code just by telling it that there's a problem, or to identify any problems. > it's not clear if GPT will have sufficient attention to handle the more complex cases. I think this could be helped with some tooling, by giving it smaller pieces of code to digest, in a new prompt and a somewhat smaller prompt. These LLM can't seem to dive into cracks if th…
> just by telling it that there's a problem how will you identify that there's a problem?
I don't see any iterations. I can sometimes get it to improve the code just by telling it that there's a problem, or to identify any problems. > it's not clear if GPT will have sufficient attention to handle the more complex cases. I think this could be helped with some tooling, by giving it smaller pieces of code to digest, in a new prompt and a somewhat smaller prompt. These LLM can't seem to dive into cracks if th…
I don't see any iterations. I can sometimes get it to improve the code just by telling it that there's a problem, or to identify any problems. > it's not clear if GPT will have sufficient attention to handle the more complex cases. I think this could be helped with some tooling, by giving it smaller pieces of code to digest, in a new prompt and a somewhat smaller prompt. These LLM can't seem to dive into cracks if th…
Do you have access to the code interpreter alpha? You can upload larger chunks of code and have it perform different tasks. One of the interesting features is that chatgpt will sometimes have some issues with its own code (python) and try to automatically correct itself.
Earlier quoted context omitted.
Not the creator, but as it's GitHub, feel free to repeat the experiment and submit a pull request with a better method; peer review.
Not everyone has the time or expertise to create a pull request. But there are issues allowed on this repo! Create an Issue if you'd like the author to address something, in my opinion.
The conclusions start from https://github.com/E-xyza/Exonerate/blob/master/bench/report... This is particularly impressive for Elixir, which is not a language that is a particular focus of GPT-4. I imagine the accuracy for Python is extremely good. Maybe near perfect for this kind of benchmark if allowed to see error messages.
It's also possible GPT-4 is better at writing Elixir since there are less beginners/students writing Elixir code and polluting the training data with bad practice or faulty code.
I don't see any iterations. I can sometimes get it to improve the code just by telling it that there's a problem, or to identify any problems. > it's not clear if GPT will have sufficient attention to handle the more complex cases. I think this could be helped with some tooling, by giving it smaller pieces of code to digest, in a new prompt and a somewhat smaller prompt. These LLM can't seem to dive into cracks if th…
> just by telling it that there's a problem how will you identify that there's a problem?
But that's actually not what I meant. You can often just tell it "there's a problem, please fix it", or "do you see any problems" and it will be able to identify it without additional input. There no requirement that you've identified a problem, it's more of a "double check that" type of prompt.