Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

51–60 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#51

Post by someplaceguy »

devin wrote: post #43
TuringNYC wrote: post #41

Earlier quoted context omitted.

>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!

It takes like 2-3 experiences of receiving a confidently wrong answer to downgrade your usage. If you use a refactoring tool to rename and it misses one, you won’t use it again.

While that would likely be my experience with a refactoring tool (unless I didn't have a better alternative), that's not my experience with ChatGPT 4. And that's considering I have very little tolerance for buggy software.

There was a period of a few weeks or months in which it seemed like ChatGPT had really degraded to the point of being unusable (although it could have been my biases). However, it seems to be better now (again, my subjective experience).

Sometimes I still catch it making really basic mistakes, but most times I can convince it to correct the mistake (especially if I point them out).

But what's most amazing to me is how ChatGPT is absolutely brilliant at some things, and not just technical or even obscure topics.

Recently, it gave me the most amazing idea for navigating a complex and nuanced social situation I was having difficulty with. And given the constraints of the situation, there was no way I could have gotten that idea otherwise, especially in the allotted time.

So despite its flaws and mistakes, I still find it to be a tremendously useful tool, even if only to point me in the right direction.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#52

Post by esperent »

TuringNYC wrote: post #41
rushingcreek wrote: post #33

Earlier quoted context omitted.

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!

I'm aware of at least one study by Stanford. PDF paper linked in this article:

https://www.techopedia.com/is-gpt-4-a-flop

Of course, I'd like to see more than one study. But this one is by a well known university, and it's pretty conclusive. GPT-4 is getting worse (especially for code, maths, and analytical reasoning) and more censored.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#53

Post by refulgentis »

rushingcreek wrote: post #44
refulgentis wrote: post #40

Earlier quoted context omitted.

There weren't any serious examples of degradation. Does only GPT-4 have to suffer a penalty for HumanEval leaking into training data/RLHF data? Ignoring those concerns, it fails a reaonable-ness smell test: We'd have to pretend its the original GPT-4 release from March 2023 until GPT-5 comes out, and only then can OpenAI's work be compared to LLAMA-2 to LLAMA-N.

There's a couple of things here: 1. I'm not saying we have to wait until GPT-5, we just need an apples-to-apples comparison where contamination is taken into account 2. GPT-4 does not seem to have improved on real-world coding tasks since March, so it's unclear where any purported HumanEval gains could've come from 3. I've personally noticed degradation anecdotally in the GPT-4 June update vs. the original March rele…

1. TL;DR: OpenAI must verify HumanEval data wasn't used in training in order to compare it?

2. Link in the post you replied to.

3. Subjectivity is fine by me! There's a motte & bailey flavor to it if we combine your comment and this one, c.f. "This is why we use the official numbers."

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#54

Post by RossBencina »

Jagerbizzle wrote: post #23
tyhjnntyny wrote: post #21

This stuff is accelerating it alarming pace. I know of three or four off the shelf solutions to self host and run models now. I have learned so much and continue to learn about how this stuff works.

Any recommendations for learning material?

TheBloke discord

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#55

Post by redox99 »

syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

I actually tried the 4 bit quants (Q4_K_M) and was a bit unimpressed. Switching to Q6_K made a huge difference, but it doesn't fit on my 3090 so it was very slow. And testing on perplexity's website which I presume is fp16 seemed even better, although that might be mostly due to sampler/prompt differences.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#58

Post by chpatrick »

smoldesu wrote: post #46
rightbyte wrote: post #35

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

Facebook has had an AI research lab for the better half of a decade now. Their open release model worked well in the past and it seems to be working well with Llama too. That looks like the extent of it. We should expect more FAANG-sized companied to do this, frankly. It's surprising to see people antipate a 'gotcha' considering how influental and transparent Pytorch was.

Given that LLMs might be a generation-defining technology it's certainly weird that one of the biggest public companies is giving theirs away for free.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#59

Post by rushingcreek »

redox99 wrote: post #57

Could you please add the prompt format to the model card, or at least here?

The model card is intended to help reproduce the HumanEval results, which we've achieved with no prompt.

However, the model is instruction-tuned to follow completions (not chats). Simply tell it what you want and it should work.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#60

Post by dragonwriter »

chpatrick wrote: post #58
smoldesu wrote: post #46

Earlier quoted context omitted.

Facebook has had an AI research lab for the better half of a decade now. Their open release model worked well in the past and it seems to be working well with Llama too. That looks like the extent of it. We should expect more FAANG-sized companied to do this, frankly. It's surprising to see people antipate a 'gotcha' considering how influental and transparent Pytorch was.

Given that LLMs might be a generation-defining technology it's certainly weird that one of the biggest public companies is giving theirs away for free.

Web browsers were also a generation-defining technology.
Post reply on HN