Code Llama, a state-of-the-art large language model for coding • Hacker News Board

Re: Code Llama, a state-of-the-art large language model for coding

#451

Post by selfhoster11 » Fri, Aug 25, 2023, 7:07 AM UTC

stuckinhell wrote: post #206

syntaxing wrote: post #185

TheBloke doesn’t joke around [1]. I’m guessing we’ll have the quantized ones by the end of the day. I’m super excited to use the 34B Python 4 bit quantized one that should just fit on a 3090. [1] https://huggingface.co/TheBloke/CodeLlama-13B-Python-fp16

What kind of cpu/gpu power do you need for quantization or these new gguf formats ?

I can quantize models up to 70B just fine with around 40-50 GB of system RAM, using the GGMLv3 format.

GGUF seems not optimised yet, since quantizing with a newer version of llama.cpp supporting the format fails on the same hardware. I expect that to be fixed shortly.

For inference, I understand that the hardware requirements will be identical as before.

Re: Code Llama, a state-of-the-art large language model for coding

#452

Post by m00nsome » Fri, Aug 25, 2023, 7:22 AM UTC

Why do they not release the unnatural Variant of the model? According to the paper it beats all of the other variants and seems to be close to GPT-4.

Re: Code Llama, a state-of-the-art large language model for coding

#453

Post by smcleod » Fri, Aug 25, 2023, 7:30 AM UTC

sean_flanigan wrote: post #425

adamgordonbell wrote: post #83

Earlier quoted context omitted.

Solutions exist that feed LLMS ctags, and seem to work well. The function signatures and symbols names for a code base are much smaller than the actual code.

I know about https://github.com/paul-gauthier/aider . Have you got a link to any others?

I'm using this right now, but it's noted that "ctags only work with GPT4" so I'm yet to get them working with llama locally.

Re: Code Llama, a state-of-the-art large language model for coding

#454

Post by easygenes » Fri, Aug 25, 2023, 7:54 AM UTC

turnsout wrote: post #387

someplaceguy wrote: post #369

Earlier quoted context omitted.

Yeah, but yours was generated by the "post unoptimized code to HN and wait for someone to optimize it" model, which, although free and doesn't require a GPU, is a much slower model.

Someone should turn this into a product! You highlight the code you want to optimize, and it posts it to hn as a semi-contextually-appropriate comment to invite code golfing, and the highest rated reply gets posted back to your repo as a PR.

What are some existing data source that are somewhat analogous to this? e.g., Project Euler.

Re: Code Llama, a state-of-the-art large language model for coding

#455

Post by KingOfCoders » Fri, Aug 25, 2023, 8:10 AM UTC

Any performance tests? (e.G. tokens/s on a 4090?)

Re: Code Llama, a state-of-the-art large language model for coding

#456

Post by KingOfCoders » Fri, Aug 25, 2023, 8:15 AM UTC

ttul wrote: post #130

mymac wrote: post #122

Never before in the history of mankind was a group so absolutely besotted with the idea of putting themselves out of a job.

That’s just one perspective… Another perspective is that LLMs enable programmers to skip a lot of the routine and boring aspects of coding - looking up stuff, essentially - so they can focus on the fun parts that engage creativity.

One coachman to the other: "Another perspective about this car thing, you can skip all the routine and boring trips - they are done with cars. You can focus on the nice trips that make you feel good".

Re: Code Llama, a state-of-the-art large language model for coding

#457

Post by timjver » Fri, Aug 25, 2023, 8:23 AM UTC

tuukkah wrote: post #445

kilolima wrote: post #417

Earlier quoted context omitted.

For printing the first 10 prime numbers, there's a one line solution to this problem: print("1, 2, 3, 5, 7, 11... and so on!

That can't be, because primes are numbers greater than 1.

They said nothing about not printing any non-primes.

Re: Code Llama, a state-of-the-art large language model for coding

#458

Post by FrozenSynapse » Fri, Aug 25, 2023, 8:35 AM UTC

daemonologist wrote: post #236

Works nearly out of the box with llama.cpp, which makes it easy to try locally: https://github.com/ggerganov/llama.cpp/issues/2766 Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt): # prints the first ten prime numbers def print_primes(): i = 2 num_printed = 0 # end of prompt while num_printed It will be interesting to see how the larger models perform, especially afte…

how much VRAM do you need to run quantised 7b model?

Re: Code Llama, a state-of-the-art large language model for coding

#459

Post by FrozenSynapse » Fri, Aug 25, 2023, 8:40 AM UTC

selfhoster11 wrote: post #450

ohyes wrote: post #121

Earlier quoted context omitted.

What hardware do you have that lets you run 7b and do other stuff at the same time?

A 7B model at 8-bit quantization takes up 7 GB of RAM. Less if you use a 6-bit quantization, which is nearly as good. Otherwise it's just a question of having enough system RAM and CPU cores, plus maybe a small discrete GPU.

how's the generation speed on CPU?

Re: Code Llama, a state-of-the-art large language model for coding

#460

Post by tuukkah » Fri, Aug 25, 2023, 8:43 AM UTC

timjver wrote: post #457

tuukkah wrote: post #445

Earlier quoted context omitted.

That can't be, because primes are numbers greater than 1.

They said nothing about not printing any non-primes.

I suppose it's indeed possible to read the assignment as "print whatever you like as long as it includes 10 or more primes" but I'd fail you for that based on not being able to communicate with humans.