Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

151–160 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#151

Post by oezi »

mhh__ wrote: post #48
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

(Chat)GPT-4s practical coding abilities are now 100x because it can code, run the code, and reason about its performance mid-response. They must be using fine tunes for this so the overall model could well be better too

Only python though, right?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#152

Post by EvgeniyZh »

esperent wrote: post #52
TuringNYC wrote: post #41

Earlier quoted context omitted.

>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!

I'm aware of at least one study by Stanford. PDF paper linked in this article: https://www.techopedia.com/is-gpt-4-a-flop Of course, I'd like to see more than one study. But this one is by a well known university, and it's pretty conclusive. GPT-4 is getting worse (especially for code, maths, and analytical reasoning) and more censored.

On most of their tests gpt-4 is not actually worse [1]. In particular coding results are affected by changed due to different output format rather than worse abilities [2]. But that's ok because the message of the paper is that there is strong drift between versions and developers should be aware of it, not that gpt becomes worse [3].

[1] https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-tim...

[2] https://twitter.com/Si_Boehm/status/1681801371656536068

[3] https://twitter.com/matei_zaharia/status/1681805357516210177

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#154

Post by krasin »

whimsicalism wrote: post #112
krasin wrote: post #36

Earlier quoted context omitted.

One possible explanation is to make PyTorch victory over TensorFlow/JAX even more apparent: with Google ceasing to release interesting open-source models, and Facebook / Meta continuing doing so (even if under a less permissive license than usual), it means that there's nothing interesting in the TensorFlow/JAX ecosystem, and a lot in the PyTorch one. (context: I work for Google DeepMind, opinions are my own)

Hmm, but I feel like Jax/Pytorch interop is only getting better. Tensorflow abstraction has already firmly lost.

> Hmm, but I feel like Jax/Pytorch interop is only getting better

There's definitely some convergence. It's good. It still stands for a fact that most people do/think about PyTorch nowadays, not JAX (speaking as someone who likes JAX).

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#155

Post by EvgeniyZh »

rushingcreek wrote: post #33
agnokapathetic wrote: post #29

GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921 https://github.com/getcursor/eval

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

I have a several arguments why contamination is probably not the main reason of performance difference.

When we worked on StarCoder, people ran gpt-4 on MultiPL-E, which doesn't have canonical solutions in the internet, and the performance was higher that what you would expect from official numbers

Official contamination analysis shows only minor drop in performance even though contamination is fairly high (you may argue that contamination is higher now or that rlhf has stronger effect)

There is significant drop in performance when testing on HumanEval+ [1], which shouldn't happen if model has canonical solutions.

BTW why don't you use HumanEval+?

[1] https://arxiv.org/abs/2305.01210

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#156

Post by jiggawatts »

haxiel wrote: post #133
rightbyte wrote: post #35

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

A previous post here on HN [1] discussed the leak of the original LLaMA model and Stable Diffusion. The article points out that the open-source community's effort resulted in more innovation, and an ecosystem built up around these models leading to rapid adoption. That gives more control to the original authors of the model against their competitors. I think Meta has recognized this and wants to keep their lead going…

They’re end-users of AI, they’re not vendors selling AI. Giving it away for free is not a loss to them.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#157

Post by modeless »

esperent wrote: post #136
modeless wrote: post #104

Earlier quoted context omitted.

Isn't this the study that asked a bunch of questions with the same answer ("yes") and basically the old model always answered "yes" and the new model always answered "no"? That's not a degradation in performance. It was never answering the questions in the first place, just guessing. The only thing that changed was the default guess.

No it's not. Here's a link to the study, you can check the questions they asked. https://arxiv.org/pdf/2307.09009.pdf

You are wrong. What I described is exactly what they did in the "math" benchmark of v1 of the study (https://arxiv.org/pdf/2307.09009v1.pdf). They asked "is this number prime" for a bunch of prime numbers. The old version gave a bunch of "reasoning" that was actually faulty and then guessed "yes". The new version guessed "no" (which is arguably a better guess as to whether a random number is prime). In neither case did it actually do the work required to answer the question and the change in "correct" answers is an illusion.

In the programming category the newer GPT-4 actually performed significantly better but started formatting code with backticks that the study's evaluation code didn't handle properly, so they falsely concluded that it was worse. https://twitter.com/Si_Boehm/status/1681801371656536068

They later submitted a revision to the study attempting to correct these blatant issues but trusting their work is clearly a terrible idea. The study was executed very poorly and should be ignored with extreme prejudice.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#158

Post by BoorishBears »

esperent wrote: post #136
modeless wrote: post #104

Earlier quoted context omitted.

Isn't this the study that asked a bunch of questions with the same answer ("yes") and basically the old model always answered "yes" and the new model always answered "no"? That's not a degradation in performance. It was never answering the questions in the first place, just guessing. The only thing that changed was the default guess.

No it's not. Here's a link to the study, you can check the questions they asked. https://arxiv.org/pdf/2307.09009.pdf

At this point you've gotten like 3 or 4 replies explaining how at best you're drawing a flawed conclusion from the paper, and at worst it's a flawed paper in itself.

Funnily enough, just skimming through it again I found yet another glaring mistake they made: they left the system prompt empty for both checkpoints, yet the headline feature of the new checkpoint was improved steerability via the system prompt: https://openai.com/blog/function-calling-and-other-api-updat...

Every time I look at this paper my inclination drifts further away from harmless incompetence. Matei Zaharia is the CTO of Databricks, it feels like too perfect of a coincidence that someone who built a career on ML and study would suddenly drop the ball right as their company is trying to pivot to on-premise MLOps, who's prime competition is ChatGPT...

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#159

Post by ddalex »

praveenhm wrote: post #140
jmorgan wrote: post #110

Earlier quoted context omitted.

It really is good. Surprisingly it seems to answer instruct-like prompts well! I’ve been using it with Ollama ( https://github.com/jmorganca/ollama ) with prompts like: ollama run phind-codellama "write c code to reverse a linked list" To run this on an m1 Mac or similar machine, you'll need around 32GB of memory for the 4-bit quantized version since it's a 34B parameter model and is quite big (20GB).

Is reverse a linked list is still a popular question on interview?

I ask it as a warmup question, I expect it to be done in 5-10 minutes.

Then comes the real question, which is "let's write fizz buzz so it generates at above 55Gbytes/second".

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#160

Post by albert_e »

PeterStuer wrote: post #141
behnamoh wrote: post #17

Earlier quoted context omitted.

Why would it? Do you know how much it costs to finetune one of these models for such a niche language? I'm not just talking about the cost of training, but also the cost of acquiring data because there's much less data about niche languages.

96 x A100 hours for a finetune according to the article. The cost of the dataset curation for a given language is hard to quantify as there are many unknowns. However, it seems perfectly crowdsourcable to volunteers.

A project like SETI@home should help these efforts I believe?
Post reply on HN