Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

291–300 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#291

Post by xpe »

someplaceguy wrote: post #289
xpe wrote: post #288

Earlier quoted context omitted.

Anecdotes sometimes take a beating, but I happen to like the personal ones. Thanks for sharing. A quick thought about your success: ChatGPT's imprecision and stochasticity can work in its favor for many creative efforts. Unexpected token connections can have a lot of value in a space where vast numbers of novel directions are worthwhile. For me, having spent thousands of hours thinking about statistics, ML, logic, an…

I don't think I've spent nearly as much time as you thinking about these things and I'm not entirely sure I understood your perspective, but I have a couple of reflections for you which perhaps you can comment on: > By themselves, sequence to sequence models are not computationally capable of deductive reasoning with an arbitrary number of steps, since that would require recursion (or iteration). Isn't the fact that…

> That said, something that I think could really make LLMs take a big step forward would be to have something akin to long-term memory.

Yes. There is significant work in this direction.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#292

Post by M4v3R »

coder543 wrote: post #245
M4v3R wrote: post #194

Earlier quoted context omitted.

A 4060 Ti with 16GB memory costs around $700-800 over here, can I connect two of these to get 32 GB memory to run the bigger models (13B stock or 30B quantized)?

One additional caveat worth considering... a lot of LLM computation is often memory bandwidth bound. The 4060 Ti is infamous for how deeply Nvidia cut the memory system. The 3060 Ti had a 256-bit bus that was capable of 448GB/s of memory bandwidth. The 4060 Ti has a paltry 128-bit bus that can only do 288GB/s, about 64% of the memory bandwidth of the previous generation. For comparison, an RTX 3090 has 935GB/s of mem…

Thanks for the info!

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#293

Post by eshack94 »

bddppq wrote: post #167
rushingcreek wrote: post #88

Earlier quoted context omitted.

I see you're using [INST] tokens. Please don't do this -- this model was not trained in this format or to be a chat model. Instead, it should be treated as a completions model (like text-davinci-003) and no system prompt should be provided. Just tell it what you want.

Thanks for pointing out. We have now updated the prompt template to follow the format.

Hey, just an FYI, phind-codellama-34b-v2 was just released, which performs even better! Just letting you know in case you wanted to add the new model to the lepton chat you guys set up.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#294

Post by eshack94 »

ccding wrote: post #86

Glad to see open source models becoming competitive and thanks for sharing the model to community! Lepton AI has hosted this model here free for everyone to give it a try (and compare to the original CodeLlama 34B version): https://codellama.lepton.run API is also available at https://codellama.lepton.run/api/v1 (it's fully compatible with OpenAI's api, so just switching the `api_base` to the this url and all your ex…

Hey, just an FYI, phind-codellama-34b-v2 was just released, which performs even better! Just letting you know in case you wanted to add the new model too!

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#296

Post by nomand »

api wrote: post #207
syntaxing wrote: post #42

I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on consumer hardware (single 3090 in this case) and did not feel like a toy. I purposely broke some functions, it fixed it. I asked some complex questions and it answered it well. I’m excited for what’s to come. I wish there was a Phind instruct mode…

Now we just need a llama.cpp VSCode plug-in.

Ollama for mac + https://continue.dev/. Otherwise c.d has hooks for other types of installs.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#299

Post by eshack94 »

jmorgan wrote: post #166
codetrotter wrote: post #163

Earlier quoted context omitted.

How do you guys get it to run? I tried with ollama, installed from Homebrew, on my M1 Max with 64GB RAM. I downloaded the phind-codellama model using ollama pull phind-codellama But when I give it a prompt, for example ollama run phind-codellama "write a production grade implementation of Sieve of Eratosthenes in Rust" It prints the following error message Error: Post "http://localhost:11434/api/generate": EOF and ex…

Yes you'll need the latest version (0.0.16) to run the 34B model. It should run great on that machine! The download (both as a Mac app and standalone binary) is available here: https://github.com/jmorganca/ollama/releases/tag/v0.0.16 . And I will work on getting that brew formula updated as well! Sorry to see you hit an error!

Is there any chance you can add the phind-codellama-34b-v2 model? Or if there's already a way to run that with ollama, can you tell me how or point me to some docs? Thanks a ton!

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#300

Post by larodi »

syntaxing wrote: post #210
larodi wrote: post #184

Earlier quoted context omitted.

A lot of things are being getting right if you look at the issues at ggerganov’s repo. To say anything as general as ‘the new file format is broken’ just means you either don’t understand the project basics or do not follow closely the commits.

So? Doesn’t mean that the moment we are using it, the format isn’t broken. I didn’t say that it wouldn’t be fixed in the future. The reality is, the current 4 bit GGUF are giving us subpar results compared to other quantization method. It’s not a helpful comment telling me that “I don’t understand the basic” rather than telling me the exact flags we should using or it’s being fixed.

Inference with llama.cop is not trivial and I can’t summarise in one post all of them parameters. What I’m saying is that in my opinion. is wrong to assume that changing from one transport to the other is causing degradation.

Llamacpp underwent some major changes last few weeks. And following the commits it took few days to stabilise. Try now , works as bliss. And compared to other inference engines such as tinygrad - is much more versatile in options how to be run.

Post reply on HN