Does this page have a bunch of blue-text runs that aren't links?
Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
11–20 of 306 posts
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#12Are you planning to switch to programming-language optimized model inside Phind? So, if a user is asking for something related to Python, that the python-optimized model gets used? If so: The Object Pascal language is completely out of fashion, and the most non-hyped language there is. However, there are hundreds of thousands of active users of Delphi, FreePascal and Lazarus. And due to the language being stable for…
> I would be willing to internally pay for the work What kind of budget do you think this will require?
Not much, I guess. It's basically writing some scripts that will take the code base of some of the available high quality pascal projects, and then depending on what is available extract/merge documentation available as PDF, PasDoc, RTF, .HLP or method/function source code comments.
I would assume that one of my devs could write the needed scripts in three weeks or so.
So, basically a budget of For me - due to missing competence - the actual challenge would be to get a sample on how training data should optimally look like (for example the Python training set), and someone doing the actual training. For a newbie to get up the required level of competence surely will take more than three weeks.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#13what hardware could one run this locally on? probably need 3090 etc right?
finally a reason to upgrade from my m1 max
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#14Are you planning to switch to programming-language optimized model inside Phind? So, if a user is asking for something related to Python, that the python-optimized model gets used? If so: The Object Pascal language is completely out of fashion, and the most non-hyped language there is. However, there are hundreds of thousands of active users of Delphi, FreePascal and Lazarus. And due to the language being stable for…
Yes, this is the direction we're heading towards! We're building a mixture of experts of different coding models that we will deploy for precisely this use case.
Nice!
I suppose that Pascal is not on your planned list of supported languages, right?
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#15Does this page have a bunch of blue-text runs that aren't links?
Yes. I think it's just a highlight. The HTML doesn't look like it's trying to be a link or anything.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#16> proprietary dataset
This reads more like advertising copy than an HN article.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#17Earlier quoted context omitted.
Yes, this is the direction we're heading towards! We're building a mixture of experts of different coding models that we will deploy for precisely this use case.
Nice! I suppose that Pascal is not on your planned list of supported languages, right?
Why would it? Do you know how much it costs to finetune one of these models for such a niche language? I'm not just talking about the cost of training, but also the cost of acquiring data because there's much less data about niche languages.
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#18Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#19Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the problems themselves aren’t really indicative of actual software engineering problems. Also, we’ve been able to get around 85% pass@1 on GPT-4 internally as of a couple weeks ago. It’s hard to say if they’ve contaminated the models with RLHF though. It still is exciting how close we’re getting with open source models but we’ve still got a decent amount of work to go!
Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B
#20Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the problems themselves aren’t really indicative of actual software engineering problems. Also, we’ve been able to get around 85% pass@1 on GPT-4 internally as of a couple weeks ago. It’s hard to say if they’ve contaminated the models with RLHF tho…
One thing I’d be curious to see is how well this translates to things outside of HumanEval! How does it compare to using ChatGPT for example.