Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

41–50 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#41

Post by TuringNYC »

rushingcreek wrote: post #33
agnokapathetic wrote: post #29

GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921 https://github.com/getcursor/eval

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

>> given all of the examples and anecdotes about degradation.

How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#42

Post by syntaxing »

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct model for me to play with. Text completion hasn’t really been all that useful for my use case.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#43

Post by devin »

TuringNYC wrote: post #41
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!

It takes like 2-3 experiences of receiving a confidently wrong answer to downgrade your usage. If you use a refactoring tool to rename and it misses one, you won’t use it again.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#44

Post by rushingcreek »

refulgentis wrote: post #40
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

There weren't any serious examples of degradation. Does only GPT-4 have to suffer a penalty for HumanEval leaking into training data/RLHF data? Ignoring those concerns, it fails a reaonable-ness smell test: We'd have to pretend its the original GPT-4 release from March 2023 until GPT-5 comes out, and only then can OpenAI's work be compared to LLAMA-2 to LLAMA-N.

There's a couple of things here:

1. I'm not saying we have to wait until GPT-5, we just need an apples-to-apples comparison where contamination is taken into account

2. GPT-4 does not seem to have improved on real-world coding tasks since March, so it's unclear where any purported HumanEval gains could've come from

3. I've personally noticed degradation anecdotally in the GPT-4 June update vs. the original March release

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#46

Post by smoldesu »

rightbyte wrote: post #35

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

Facebook has had an AI research lab for the better half of a decade now. Their open release model worked well in the past and it seems to be working well with Llama too. That looks like the extent of it.

We should expect more FAANG-sized companied to do this, frankly. It's surprising to see people antipate a 'gotcha' considering how influental and transparent Pytorch was.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#47

Post by rushingcreek »

xmly wrote: post #45

What would the hardware requirement for fine-tuning a Llama 2 ?

We didn't want to use LoRA to maximize quality, so we used 32 A100-80GB with a sequence length of 4096. It's possible to do a native fine-tune on as little as 8 A100-80GB with DeepSpeed Zero 3, but it will take longer.

With LoRA you can probably get away with just a few 4090s.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#48

Post by mhh__ »

rushingcreek wrote: post #33
agnokapathetic wrote: post #29

GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921 https://github.com/getcursor/eval

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

(Chat)GPT-4s practical coding abilities are now 100x because it can code, run the code, and reason about its performance mid-response. They must be using fine tunes for this so the overall model could well be better too

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#49

Post by rushingcreek »

syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

Our models should handle instructions reasonably well. We're working on setting up a hosted Huggingface space to make it easier to play with them and we'll also set up a hosted "Phind chat" mode for these models.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#50

Post by disconcision »

tomr75 wrote: post #13

what hardware could one run this locally on? probably need 3090 etc right? finally a reason to upgrade from my m1 max

maybe worth noting that depending on specifics a 3090 isn't necessarily a strict upgrade over an M1 max. While a 3090 will be faster on any model it can run, an M1 with 32GB of ram or more can load models which a 3090 cannot
Post reply on HN