> proprietary dataset This reads more like advertising copy than an HN article.
Can you suggest better terminology? It seems normal to me.
Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
31–40 of 306 posts
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#32That’s good and all but GPT4 can do all languages, not just 2-3
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#33GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921 https://github.com/getcursor/eval
Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4.
It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation.
This is why we use the official numbers.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#34what hardware could one run this locally on? probably need 3090 etc right? finally a reason to upgrade from my m1 max
34B is basically an exact fit for a 3090, when quantized to 4 bits.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#35Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#36Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?
One possible explanation is to make PyTorch victory over TensorFlow/JAX even more apparent: with Google ceasing to release interesting open-source models, and Facebook / Meta continuing doing so (even if under a less permissive license than usual), it means that there's nothing interesting in the TensorFlow/JAX ecosystem, and a lot in the PyTorch one.
(context: I work for Google DeepMind, opinions are my own)
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#37Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?
Phind did the fine-tuning, not Facebook.
This is actually an excellent demonstration of a good reason to open source something. Another company immediately made a dramatic improvement.
Now anyone from Facebook or any companies they have invested in can benefit from that improvement and information about how they did it. Facebook could even take this exact model and add a coding assistant into Messenger if they wanted (not that they are interested in that). Or apply it to other things.
OpenAI has a big lead. The most effective counter to that is actually to team up.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#38Ive used GPT 4 for pretty much all of my programming needs and the convenience of a 20 dollar subscription taking care of everything and letting me use a LLM without having to set up any models or servers has been just so simple, is the 2 percent gain worth looking into running a local model again? I tried running a local model a couple months ago but the perfomance was bad. I know code llama came out very recently b…
Personally I doubt that GPT-4 is really still at 67%.
I would love to see some head-to-head examples.
Is anyone hosting this somewhere that can be accessed for free right now?
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#39This stuff is accelerating it alarming pace. I know of three or four off the shelf solutions to self host and run models now. I have learned so much and continue to learn about how this stuff works.
Any recommendations for learning material?
The localllama subreddit has lots of content though mostly unstructured
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#40GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921 https://github.com/getcursor/eval
Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.
There weren't any serious examples of degradation.
Does only GPT-4 have to suffer a penalty for HumanEval leaking into training data/RLHF data?
Ignoring those concerns, it fails a reaonable-ness smell test:
We'd have to pretend its the original GPT-4 release from March 2023 until GPT-5 comes out, and only then can OpenAI's work be compared to LLAMA-2 to LLAMA-N.