Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

91–100 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#91

Post by dontupvoteme »

mhh__ wrote: post #48
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

(Chat)GPT-4s practical coding abilities are now 100x because it can code, run the code, and reason about its performance mid-response. They must be using fine tunes for this so the overall model could well be better too

You can do that as well, under your complete control. That's a framework they put around the model.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#92

Post by gmm1990 »

syntaxing wrote: post #79
gmm1990 wrote: post #74

Earlier quoted context omitted.

What parameters/prompts did you use, I was able to get ok results but not comparable to chat gpt or even bard

I used textgen, instruct mode with LLaMa 2 template, “simple” parameters. What quantization method are you using?

I used the llama.cpp quantize q_0 option. Seemed to vary widely just adjusting the temperature parameters. Was giving it an example db query saving to file in json and asking create a similar function did multiple queries.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#93

Post by [deleted] »

mensetmanusman wrote: post #63
rvz wrote: post #61

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…

Assuming more compute is better, free models will always be a step behind in speed/quality and a step ahead in privacy.

[deleted]

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#94

Post by [deleted] »

ShamelessC wrote: post #77
rvz wrote: post #61

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…

Meta isn't being "heroic" or "winning a long fought battle" or whatever analogy you want to use in order to prop them up and make them look better than they are. Do I appreciate the free weights from them? Yes. Did they contribute to a genocide in Myanmar? Also yes. Take the weights and move on. They don't deserve the excessive praise. They aren't a rock band. They are an advertising company.

[deleted]

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#97

Post by pclmulqdq »

refulgentis wrote: post #53
rushingcreek wrote: post #44

Earlier quoted context omitted.

There's a couple of things here: 1. I'm not saying we have to wait until GPT-5, we just need an apples-to-apples comparison where contamination is taken into account 2. GPT-4 does not seem to have improved on real-world coding tasks since March, so it's unclear where any purported HumanEval gains could've come from 3. I've personally noticed degradation anecdotally in the GPT-4 June update vs. the original March rele…

1. TL;DR: OpenAI must verify HumanEval data wasn't used in training in order to compare it? 2. Link in the post you replied to. 3. Subjectivity is fine by me! There's a motte & bailey flavor to it if we combine your comment and this one, c.f. "This is why we use the official numbers."

I think you're assuming that OpenAI is incentivized to benchmark honestly. Like every other company for which a benchmark is a goal, they are not.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#98

Post by sitkack »

behnamoh wrote: post #17
Fischgericht wrote: post #14

Earlier quoted context omitted.

Nice! I suppose that Pascal is not on your planned list of supported languages, right?

Why would it? Do you know how much it costs to finetune one of these models for such a niche language? I'm not just talking about the cost of training, but also the cost of acquiring data because there's much less data about niche languages.

> Do you know how much it costs to finetune

Between 30-3000$, often in the 300$ range.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#99

Post by xpe »

someplaceguy wrote: post #51
devin wrote: post #43

Earlier quoted context omitted.

It takes like 2-3 experiences of receiving a confidently wrong answer to downgrade your usage. If you use a refactoring tool to rename and it misses one, you won’t use it again.

While that would likely be my experience with a refactoring tool (unless I didn't have a better alternative), that's not my experience with ChatGPT 4. And that's considering I have very little tolerance for buggy software. There was a period of a few weeks or months in which it seemed like ChatGPT had really degraded to the point of being unusable (although it could have been my biases). However, it seems to be bette…

> So despite its flaws and mistakes, I still find it to be a tremendously useful tool, even if only to point me in the right direction.

Much of this resonates. That said, I get tremendous value simply by writing things down (or dictating them) and replying to my own question. I would expect that a sizable fraction of people have forgotten about these strategies and/or don't use them when they are most useful. For many, there is tremendous muscle memory to run a Hooli search almost on mental autopilot. Who has time to slow down and write a well-conceived question? Or perhaps we should turn it around ... On a longer time horizon, who would want to waste time with poorly-conceived questions?

It is the question that starts the process. So we should ask good questions. Do we? I'd be curious about the usage data OpenAI collects. I do my best to lower expectations about people in general, but I'm confident I'd still be unprepared for the level of thought put into questions.

> But what's most amazing to me is how ChatGPT is absolutely brilliant at some things, and not just technical or even obscure topics.

I'm not amazed in the way you are. I expect a variation in quality across topics and domains and question styles.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#100

Post by sitkack »

Fischgericht wrote: post #8

Are you planning to switch to programming-language optimized model inside Phind? So, if a user is asking for something related to Python, that the python-optimized model gets used? If so: The Object Pascal language is completely out of fashion, and the most non-hyped language there is. However, there are hundreds of thousands of active users of Delphi, FreePascal and Lazarus. And due to the language being stable for…

Where are the troves of Pascal code? Also manuals, books, etc. The quality doesn't have to be great. You can label and generate more data once you have enough to bootstrap the model.
Post reply on HN