Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

61–70 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#61

Post by rvz »

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner.

Cloud-only models cannot compete with free and they continue to lose hundreds of thousands of dollars a day, especially OpenAI getting disrupted by the $0 free AI model ecosystem thanks to Meta.

It is clear that the REAL 'Open AI' has always been Meta. Funny how many here dismissed Meta a year ago and since the days of the ConvNet hype, they have been releasing their AI research more frequently than OpenAI is trying to update their GPT model.

For Meta, they are already at the finish line and thanks to them you have CodeLlama; accelerating and driving inflated developer salaries to the bottom. No need for expensive, tired seniors and inexperienced juniors both doing the same developer job worse than two engineers with CodeLlama who can do the same work faster for less.

The future is hiring less developers but utilising AI and not hiring more with extreme inflated salaries.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#62

Post by mensetmanusman »

devin wrote: post #43
TuringNYC wrote: post #41

Earlier quoted context omitted.

>> given all of the examples and anecdotes about degradation. How many examples and anecdotes about degradation are actually scientific side-by-side studies? I see absurd articles online about ChatGPT usage going down the drain by kids, completely failing to consider even the most basic fact of seasonality and how school is out for the summer!

It takes like 2-3 experiences of receiving a confidently wrong answer to downgrade your usage. If you use a refactoring tool to rename and it misses one, you won’t use it again.

Depends on what you expect it capable of given the limitations of these systems.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#63

Post by mensetmanusman »

rvz wrote: post #61

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…

Assuming more compute is better, free models will always be a step behind in speed/quality and a step ahead in privacy.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#64

Post by dragonwriter »

DigitalNoumena wrote: post #25
bfogelman wrote: post #19

Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the problems themselves aren’t really indicative of actual software engineering problems. Also, we’ve been able to get around 85% pass@1 on GPT-4 internally as of a couple weeks ago. It’s hard to say if they’ve contaminated the models with RLHF tho…

I think the issue of test set contamination is important, but it’s academic - when a model contains a good enough distilled representation of arguably all the code out there, does it really matter whether it can generalise OOD? Realistically how many of the practical use cases where it’ll be applied will be OOD? If you can take GPT4 there then you are either a genius or working on something extremely novel so why use…

> when a model contains a good enough distilled representation of arguably all the code out there, does it really matter whether it can generalise OOD?

If its contaminated by the test set being in the model’s training set, then the test is no longer (assuming it was in the first place) a valid measure of whether the model has “a good enough distilled representation of arguably all the code out there”.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#65

Post by thentherewere2 »

chpatrick wrote: post #58
smoldesu wrote: post #46

Earlier quoted context omitted.

Facebook has had an AI research lab for the better half of a decade now. Their open release model worked well in the past and it seems to be working well with Llama too. That looks like the extent of it. We should expect more FAANG-sized companied to do this, frankly. It's surprising to see people antipate a 'gotcha' considering how influental and transparent Pytorch was.

Given that LLMs might be a generation-defining technology it's certainly weird that one of the biggest public companies is giving theirs away for free.

They're actually not giving it away for free. At least not LLAMA v2. Once a product successfully monetizes and gains over a certain number of users, an official license from Meta has to be negotiated otherwise Meta could revoke usage of the model. At least that's roughly what the license for LLAMAv2 implies.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#66

Post by redox99 »

rushingcreek wrote: post #59
redox99 wrote: post #57

Could you please add the prompt format to the model card, or at least here?

The model card is intended to help reproduce the HumanEval results, which we've achieved with no prompt. However, the model is instruction-tuned to follow completions (not chats). Simply tell it what you want and it should work.

Yeah but I assume it has a prompt format, for example CodeLlama34B Instruct the prompt format is

[INST] Write a function in Python that prints HelloWorld [/INST]

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#67

Post by chpatrick »

thentherewere2 wrote: post #65
chpatrick wrote: post #58

Earlier quoted context omitted.

Given that LLMs might be a generation-defining technology it's certainly weird that one of the biggest public companies is giving theirs away for free.

They're actually not giving it away for free. At least not LLAMA v2. Once a product successfully monetizes and gains over a certain number of users, an official license from Meta has to be negotiated otherwise Meta could revoke usage of the model. At least that's roughly what the license for LLAMAv2 implies.

But that clause pretty much only applies to mega companies like FB no?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#68

Post by rushingcreek »

redox99 wrote: post #66
rushingcreek wrote: post #59

Earlier quoted context omitted.

The model card is intended to help reproduce the HumanEval results, which we've achieved with no prompt. However, the model is instruction-tuned to follow completions (not chats). Simply tell it what you want and it should work.

Yeah but I assume it has a prompt format, for example CodeLlama34B Instruct the prompt format is [INST] Write a function in Python that prints HelloWorld [/INST]

Nope, just ask your question. It's not chat-tuned the same way as CodeLlama34B Instruct.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#69

Post by redox99 »

rushingcreek wrote: post #68
redox99 wrote: post #66

Earlier quoted context omitted.

Yeah but I assume it has a prompt format, for example CodeLlama34B Instruct the prompt format is [INST] Write a function in Python that prints HelloWorld [/INST]

Nope, just ask your question. It's not chat-tuned the same way as CodeLlama34B Instruct.

Interesting. That seems a pretty odd choice. Without a prompt format (like almost all instruction finetunes have) the model might continue your instruction instead of answering the instruction, since it does not have a delimiter for where the instruction ends and the response begins.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#70

Post by thentherewere2 »

chpatrick wrote: post #67
thentherewere2 wrote: post #65

Earlier quoted context omitted.

They're actually not giving it away for free. At least not LLAMA v2. Once a product successfully monetizes and gains over a certain number of users, an official license from Meta has to be negotiated otherwise Meta could revoke usage of the model. At least that's roughly what the license for LLAMAv2 implies.

But that clause pretty much only applies to mega companies like FB no?

Yeah IIRC its something like 700m users. ie. meta gets to piggy back off any generational winners
Post reply on HN