Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

161–170 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#161

Post by 037 »

ahahahahah wrote: post #73
thentherewere2 wrote: post #65

Earlier quoted context omitted.

They're actually not giving it away for free. At least not LLAMA v2. Once a product successfully monetizes and gains over a certain number of users, an official license from Meta has to be negotiated otherwise Meta could revoke usage of the model. At least that's roughly what the license for LLAMAv2 implies.

That's not quite right. The license never gets revoked. Once over a certain size, the company will not receive licenses to new versions/releases.

This is how I understand it too, and it's very interesting. In any case, if your company reaches that size, I'm sure you'll have replaced it with something else by that time. I'm confident many fully open-source, high-quality and specialized models will be available in the next years - even thanks to Meta. Whatever their strategy is, the models are on our machines, legally, and this is incredible.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#162

Post by raincole »

rushingcreek wrote: post #33
agnokapathetic wrote: post #29

GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921 https://github.com/getcursor/eval

Right, but there's no contamination studies there. I suspect that RLHF data leaked HumanEval into GPT-4. It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation. This is why we use the official numbers.

But the model in OP is fine-tuned by "a proprietary dataset of ~80k high-quality programming problems and solutions". How do we know it's not contaminated by HumanEval too?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#163

Post by codetrotter »

jmorgan wrote: post #110
syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

It really is good. Surprisingly it seems to answer instruct-like prompts well! I’ve been using it with Ollama ( https://github.com/jmorganca/ollama ) with prompts like: ollama run phind-codellama "write c code to reverse a linked list" To run this on an m1 Mac or similar machine, you'll need around 32GB of memory for the 4-bit quantized version since it's a 34B parameter model and is quite big (20GB).

How do you guys get it to run?

I tried with ollama, installed from Homebrew, on my M1 Max with 64GB RAM.

I downloaded the phind-codellama model using

  ollama pull phind-codellama
But when I give it a prompt, for example

  ollama run phind-codellama "write a production grade implementation of Sieve of Eratosthenes in Rust"
It prints the following error message

    Error: Post "http://localhost:11434/api/generate": EOF
and exits.

Even though it worked to run ollama with some other models.

Is the version in homebrew not able to run phind-codellama?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#164

Post by euniceee3 »

So for huge log files would the entire thing be feed in with the offending lines of previous log files identified files in 4096 chunked tokens? Any one got a better workflow for training based on the above scenario? These log files are multiple MBs within a few minutes from the Android kernel.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#165

Post by 037 »

mensetmanusman wrote: post #63
rvz wrote: post #61

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…

Assuming more compute is better, free models will always be a step behind in speed/quality and a step ahead in privacy.

Yes, on our local machines. But on specialized remote computers, I guess it will be fast, accurate, and cheaper.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#166

Post by jmorgan »

codetrotter wrote: post #163
jmorgan wrote: post #110

Earlier quoted context omitted.

It really is good. Surprisingly it seems to answer instruct-like prompts well! I’ve been using it with Ollama ( https://github.com/jmorganca/ollama ) with prompts like: ollama run phind-codellama "write c code to reverse a linked list" To run this on an m1 Mac or similar machine, you'll need around 32GB of memory for the 4-bit quantized version since it's a 34B parameter model and is quite big (20GB).

How do you guys get it to run? I tried with ollama, installed from Homebrew, on my M1 Max with 64GB RAM. I downloaded the phind-codellama model using ollama pull phind-codellama But when I give it a prompt, for example ollama run phind-codellama "write a production grade implementation of Sieve of Eratosthenes in Rust" It prints the following error message Error: Post "http://localhost:11434/api/generate": EOF and ex…

Yes you'll need the latest version (0.0.16) to run the 34B model. It should run great on that machine!

The download (both as a Mac app and standalone binary) is available here: https://github.com/jmorganca/ollama/releases/tag/v0.0.16. And I will work on getting that brew formula updated as well! Sorry to see you hit an error!

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#167

Post by bddppq »

rushingcreek wrote: post #88
ccding wrote: post #86

Glad to see open source models becoming competitive and thanks for sharing the model to community! Lepton AI has hosted this model here free for everyone to give it a try (and compare to the original CodeLlama 34B version): https://codellama.lepton.run API is also available at https://codellama.lepton.run/api/v1 (it's fully compatible with OpenAI's api, so just switching the `api_base` to the this url and all your ex…

I see you're using [INST] tokens. Please don't do this -- this model was not trained in this format or to be a chat model. Instead, it should be treated as a completions model (like text-davinci-003) and no system prompt should be provided. Just tell it what you want.

Thanks for pointing out. We have now updated the prompt template to follow the format.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#168

Post by 037 »

robertnishihara wrote: post #80

It's amazing to see how rapidly things are moving. You can try out CodeLlama-34B on Anyscale Endpoints (an LLM inference API we're building here at Anyscale for open source LLMs). https://app.endpoints.anyscale.com/

From the FAQ:

> What rights do you claim to my queries?

> During the alpha release, we may use your inputs and outputs to improve the service. In a future release, we may provide the ability to opt out of certain data uses.

I understand that you are in alpha, but paying without any privacy guarantees is a little hard to accept right now.

Otherwise your service seems really nice, I'm sure a lot of people have been waiting for something like this. Any ETA on the opt-out and further clarification on the matter?

Thank you.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#169

Post by qup »

trickstra wrote: post #150
dado3212 wrote: post #149

Earlier quoted context omitted.

Is this really the line we want to draw in the sand? It’s not open source because it can’t be trivially used by AWS and Google? It feels like the popularism is the point, not the complete lack of restrictions.

Well if we are talking about the point of open source, then really, the lack of restrictions is the point. Some commercial entities wish to redefine the term because it benefits their marketing, but is that the line we want to draw in the sand?

You give them an inch...

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#170

Post by IanCal »

kirill5pol wrote: post #124
sitkack wrote: post #98

Earlier quoted context omitted.

> Do you know how much it costs to finetune Between 30-3000$, often in the 300$ range.

From their numbers 3 hours with 32 A100 80GBs. From lambda cloud: 3 hours * 4 ~22$/hr for 8x A100 ~= $265 So yeah not too expensive even for a native fine tune (obviously this ignores all other costs other than the GPUs)

It's wild to me that the raw compute cost is so low.
Post reply on HN