Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
1–10 of 33 posts
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#2today for me it coded in scss and yaml to write bitwise ops lol
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#3TL;DR?
(Yes, GPT-4 is prob better... but by how much and on what?) A table would've been easier
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#4TL;DR? (Yes, GPT-4 is prob better... but by how much and on what?) A table would've been easier
Not the creator, but as it's GitHub, feel free to repeat the experiment and submit a pull request with a better method; peer review.
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#5TL;DR? (Yes, GPT-4 is prob better... but by how much and on what?) A table would've been easier
Not the creator, but as it's GitHub, feel free to repeat the experiment and submit a pull request with a better method; peer review.
Not everyone has the time or expertise to create a pull request. But there are issues allowed on this repo! Create an Issue if you'd like the author to address something, in my opinion.
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#6The conclusions start from https://github.com/E-xyza/Exonerate/blob/master/bench/report...
This is particularly impressive for Elixir, which is not a language that is a particular focus of GPT-4. I imagine the accuracy for Python is extremely good. Maybe near perfect for this kind of benchmark if allowed to see error messages.
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#7Not to split hairs, but John Henry competed against a steam drill. I'm not sure what an iron track layer is...
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#8TL;DR? (Yes, GPT-4 is prob better... but by how much and on what?) A table would've been easier
> So, should you use GPT to generate your OpenAPI validations? Probably not... yet... I'm looking forward to repeating this experiment with GPT-6, and maybe GPT-7 will be able to generate an JSONSchema compiler and replace this library altogether.
from https://github.com/E-xyza/Exonerate/blob/master/bench/report...
(I believe the author is significantly underestimating the pace of progress)
Specific numbers are at https://github.com/E-xyza/Exonerate/blob/master/bench/report.... GPT-4 does significantly better.
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#9TL;DR? (Yes, GPT-4 is prob better... but by how much and on what?) A table would've been easier
You just couldn't scroll to the bottom of the page, eh?
The conclusion is that neither 3.5 nor 4 are good enough because for anything none trivial they generate code that is often subtly wrong. Might still speed up somebody new to the language/project/learning or I would say: with additional tooling/plugins/"prompt engineering"/tinkering the author might get useful results.
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#10The conclusions start from https://github.com/E-xyza/Exonerate/blob/master/bench/report... This is particularly impressive for Elixir, which is not a language that is a particular focus of GPT-4. I imagine the accuracy for Python is extremely good. Maybe near perfect for this kind of benchmark if allowed to see error messages.
It's also possible GPT-4 is better at writing Elixir since there are less beginners/students writing Elixir code and polluting the training data with bad practice or faulty code.