Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

71–80 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#71

Post by syntaxing »

redox99 wrote: post #55
syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

I actually tried the 4 bit quants (Q4_K_M) and was a bit unimpressed. Switching to Q6_K made a huge difference, but it doesn't fit on my 3090 so it was very slow. And testing on perplexity's website which I presume is fp16 seemed even better, although that might be mostly due to sampler/prompt differences.

I have a hunch something is broken with the GGUF. I had terrible results using llama cpp as well.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#72

Post by elorant »

rightbyte wrote: post #35

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

If Facebook were to implement a language model inside the platform now it would raise a lot of privacy issues. By publishing free to use models they can rely on others to do the work of training the society into using such tools at mass scale. So by the time they adopt in in all their services there will be much less backlash.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#73

Post by ahahahahah »

thentherewere2 wrote: post #65
chpatrick wrote: post #58

Earlier quoted context omitted.

Given that LLMs might be a generation-defining technology it's certainly weird that one of the biggest public companies is giving theirs away for free.

They're actually not giving it away for free. At least not LLAMA v2. Once a product successfully monetizes and gains over a certain number of users, an official license from Meta has to be negotiated otherwise Meta could revoke usage of the model. At least that's roughly what the license for LLAMAv2 implies.

That's not quite right. The license never gets revoked. Once over a certain size, the company will not receive licenses to new versions/releases.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#74

Post by gmm1990 »

syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

What parameters/prompts did you use, I was able to get ok results but not comparable to chat gpt or even bard

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#75

Post by [deleted] »

thentherewere2 wrote: post #65
chpatrick wrote: post #58

Earlier quoted context omitted.

Given that LLMs might be a generation-defining technology it's certainly weird that one of the biggest public companies is giving theirs away for free.

They're actually not giving it away for free. At least not LLAMA v2. Once a product successfully monetizes and gains over a certain number of users, an official license from Meta has to be negotiated otherwise Meta could revoke usage of the model. At least that's roughly what the license for LLAMAv2 implies.

[deleted]

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#76

Post by rushingcreek »

gmm1990 wrote: post #74
syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

What parameters/prompts did you use, I was able to get ok results but not comparable to chat gpt or even bard

We used no prompt (only autocomplete) for the HumanEval reproduction: https://huggingface.co/Phind/Phind-CodeLlama-34B-v1.

But you should be able to get it to do stuff just by telling it what you want. Note that it's completion tuned (not chat-tuned), so it should perform better on single-turn conversations.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#77

Post by ShamelessC »

rvz wrote: post #61

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…

Meta isn't being "heroic" or "winning a long fought battle" or whatever analogy you want to use in order to prop them up and make them look better than they are.

Do I appreciate the free weights from them? Yes. Did they contribute to a genocide in Myanmar? Also yes.

Take the weights and move on. They don't deserve the excessive praise. They aren't a rock band. They are an advertising company.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#78

Post by ethbr1 »

someplaceguy wrote: post #51
devin wrote: post #43

Earlier quoted context omitted.

It takes like 2-3 experiences of receiving a confidently wrong answer to downgrade your usage. If you use a refactoring tool to rename and it misses one, you won’t use it again.

While that would likely be my experience with a refactoring tool (unless I didn't have a better alternative), that's not my experience with ChatGPT 4. And that's considering I have very little tolerance for buggy software. There was a period of a few weeks or months in which it seemed like ChatGPT had really degraded to the point of being unusable (although it could have been my biases). However, it seems to be bette…

Given the fact that OpenAI has constant resources (for any given small span of time) and varying demand (users and query type), it's not crazy to think they dynamically adjust to consume all available resources on their side.

Obviously the base model would be the same, but aren't there are +/- flavors they could overlay with extra compute? E.g. multi-pass, additional experts, etc.

The benefits to giving someone an occasional "magic" answer are too great not to.

Have there been any wide studies on same-prompt-different-times?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#79

Post by syntaxing »

gmm1990 wrote: post #74
syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

What parameters/prompts did you use, I was able to get ok results but not comparable to chat gpt or even bard

I used textgen, instruct mode with LLaMa 2 template, “simple” parameters. What quantization method are you using?
Post reply on HN