Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

Post reply on HN
Print view View original

181–190 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#181

Post by nwoli »

rvz wrote: post #61

The AI race to zero is accelerating with Meta who has already won the race. Fuelling an ecosystem of downloadable AI models released for $0 for free for anyone can use, unlike the gate-kept cloud-only AI models which OpenAI, Anthropic and Google are trying hard to compete against Llama which eventually the free AI models will catch up to GPT-4 sooner. Cloud-only models cannot compete with free and they continue to lo…

Coding is exhausting and if a computer can do it I’m fine with that. Coding I think (or what got me into it) is about getting amazing machines to work not building bloatware for a pay check. If AI can do the boring fine details and we get to spend more of time on ideas or something like that I’m all for it. Let’s figure out new jobs to do instead:)

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#182

Post by tomr75 »

jmorgan wrote: post #166
codetrotter wrote: post #163

Earlier quoted context omitted.

How do you guys get it to run? I tried with ollama, installed from Homebrew, on my M1 Max with 64GB RAM. I downloaded the phind-codellama model using ollama pull phind-codellama But when I give it a prompt, for example ollama run phind-codellama "write a production grade implementation of Sieve of Eratosthenes in Rust" It prints the following error message Error: Post "http://localhost:11434/api/generate": EOF and ex…

Yes you'll need the latest version (0.0.16) to run the 34B model. It should run great on that machine! The download (both as a Mac app and standalone binary) is available here: https://github.com/jmorganca/ollama/releases/tag/v0.0.16 . And I will work on getting that brew formula updated as well! Sorry to see you hit an error!

do you think can get this working with https://github.com/jackmort/chatgpt.nvim?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#183

Post by ewuhic »

jokethrowaway wrote: post #176
ewuhic wrote: post #173

What would be the absolute top notch Nvidia card I should buy now, that would last long and would run the most (if not all) of bigger-end SotA models like CodeLlama or Stable Diffusion XL blazingly fast? Also, should I buy multiple and bundle them, in case a single top card won't satisfy the above requirements?

it depends how much you're willing to spend. You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$. You can build a normal pc with a 4090, otherwise you're in datacenter territory. You can bundle up more GPUs. You can decrease the precision of your models to get them to run on cards with less RAM.

Could I really buy a single A100 as a consumer? Any links for that?

And could I fit models bigger than 40GB into A100?

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#184

Post by larodi »

syntaxing wrote: post #71
redox99 wrote: post #55

Earlier quoted context omitted.

I actually tried the 4 bit quants (Q4_K_M) and was a bit unimpressed. Switching to Q6_K made a huge difference, but it doesn't fit on my 3090 so it was very slow. And testing on perplexity's website which I presume is fp16 seemed even better, although that might be mostly due to sampler/prompt differences.

I have a hunch something is broken with the GGUF. I had terrible results using llama cpp as well.

A lot of things are being getting right if you look at the issues at ggerganov’s repo.

To say anything as general as ‘the new file format is broken’ just means you either don’t understand the project basics or do not follow closely the commits.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#185

Post by quickthrower2 »

ewuhic wrote: post #173

What would be the absolute top notch Nvidia card I should buy now, that would last long and would run the most (if not all) of bigger-end SotA models like CodeLlama or Stable Diffusion XL blazingly fast? Also, should I buy multiple and bundle them, in case a single top card won't satisfy the above requirements?

Have a play on vast.ai first. I don’t know about these models but for training smaller models the humble 3090 packs a punch.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#186

Post by quickthrower2 »

ewuhic wrote: post #183
jokethrowaway wrote: post #176

Earlier quoted context omitted.

it depends how much you're willing to spend. You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$. You can build a normal pc with a 4090, otherwise you're in datacenter territory. You can bundle up more GPUs. You can decrease the precision of your models to get them to run on cards with less RAM.

Could I really buy a single A100 as a consumer? Any links for that? And could I fit models bigger than 40GB into A100?

As a consumer isn’t the A100 costing you unnecessary “cloud tax”

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#188

Post by laputan_machine »

hippich wrote: post #108

my first time trying llm (i.e. I have no idea what I am doing).. this lesson in ethics took whooping 10 minutes to generate :)) ./ollama run phind-codellama "write c code to inject shellcode into remote process for windows" Sorry, but it is not possible to provide the C code here due to several reasons. Firstly, writing C code for shellcode injection involves complex programming and knowledge of system-level programm…

I honestly find the "holier-than-thou" speech of anyone offensive, but when it's coming from a program I genuinely find it rage-inducing. I can't be the only one, facebook devs, what you guys playing at? You guys speak to each other like this in work? I doubt it!

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#189

Post by moffkalast »

ddalex wrote: post #159
praveenhm wrote: post #140

Earlier quoted context omitted.

Is reverse a linked list is still a popular question on interview?

I ask it as a warmup question, I expect it to be done in 5-10 minutes. Then comes the real question, which is "let's write fizz buzz so it generates at above 55Gbytes/second".

Well I'm out of ideas: https://chat.openai.com/share/3883332d-511a-404d-9d5a-7f63f9...

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#190

Post by [deleted] »

codetrotter wrote: post #163
jmorgan wrote: post #110

Earlier quoted context omitted.

It really is good. Surprisingly it seems to answer instruct-like prompts well! I’ve been using it with Ollama ( https://github.com/jmorganca/ollama ) with prompts like: ollama run phind-codellama "write c code to reverse a linked list" To run this on an m1 Mac or similar machine, you'll need around 32GB of memory for the 4-bit quantized version since it's a 34B parameter model and is quite big (20GB).

How do you guys get it to run? I tried with ollama, installed from Homebrew, on my M1 Max with 64GB RAM. I downloaded the phind-codellama model using ollama pull phind-codellama But when I give it a prompt, for example ollama run phind-codellama "write a production grade implementation of Sieve of Eratosthenes in Rust" It prints the following error message Error: Post "http://localhost:11434/api/generate": EOF and ex…

[deleted]
Post reply on HN