This stuff is accelerating it alarming pace. I know of three or four off the shelf solutions to self host and run models now. I have learned so much and continue to learn about how this stuff works.
Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
21–30 of 306 posts
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#22Ive used GPT 4 for pretty much all of my programming needs and the convenience of a 20 dollar subscription taking care of everything and letting me use a LLM without having to set up any models or servers has been just so simple, is the 2 percent gain worth looking into running a local model again? I tried running a local model a couple months ago but the perfomance was bad. I know code llama came out very recently but does anyone have any thoughts on perfomance regardi g programming tasks compared to GPT
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#23This stuff is accelerating it alarming pace. I know of three or four off the shelf solutions to self host and run models now. I have learned so much and continue to learn about how this stuff works.
Any recommendations for learning material?
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#24Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the problems themselves aren’t really indicative of actual software engineering problems. Also, we’ve been able to get around 85% pass@1 on GPT-4 internally as of a couple weeks ago. It’s hard to say if they’ve contaminated the models with RLHF tho…
Yes -- we're being careful with our claims here. This model is not yet necessarily a better coding model overall, but it's strong on Python.
We're working hard to use these advances to make models that are production ready. One such idea is to run a mixture of experts on various fine-tuned CodeLlamas.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#25Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the problems themselves aren’t really indicative of actual software engineering problems. Also, we’ve been able to get around 85% pass@1 on GPT-4 internally as of a couple weeks ago. It’s hard to say if they’ve contaminated the models with RLHF tho…
I think the issue of test set contamination is important, but it’s academic - when a model contains a good enough distilled representation of arguably all the code out there, does it really matter whether it can generalise OOD?
Realistically how many of the practical use cases where it’ll be applied will be OOD? If you can take GPT4 there then you are either a genius or working on something extremely novel so why use GPT4 in the first place?
I understand the goal is for LLMs to get there, but the majority of practical applications just don’t need that.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#26Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the problems themselves aren’t really indicative of actual software engineering problems. Also, we’ve been able to get around 85% pass@1 on GPT-4 internally as of a couple weeks ago. It’s hard to say if they’ve contaminated the models with RLHF tho…
[deleted]
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#27This stuff is accelerating it alarming pace. I know of three or four off the shelf solutions to self host and run models now. I have learned so much and continue to learn about how this stuff works.
[deleted]
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#28Ive used GPT 4 for pretty much all of my programming needs and the convenience of a 20 dollar subscription taking care of everything and letting me use a LLM without having to set up any models or servers has been just so simple, is the 2 percent gain worth looking into running a local model again? I tried running a local model a couple months ago but the perfomance was bad. I know code llama came out very recently b…
I think for them it's about de-risking their business.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#29GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here)
https://twitter.com/amanrsanger/status/1635751764577361921 https://github.com/getcursor/eval
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#30Ive used GPT 4 for pretty much all of my programming needs and the convenience of a 20 dollar subscription taking care of everything and letting me use a LLM without having to set up any models or servers has been just so simple, is the 2 percent gain worth looking into running a local model again? I tried running a local model a couple months ago but the perfomance was bad. I know code llama came out very recently b…
If all you care about is convenience, then anything cloud or SaaS will be better for you.