Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

Post reply on HN
Print view View original

461–470 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#461

Post by maleldil »

warent wrote: post #405
maleldil wrote: post #333

Earlier quoted context omitted.

I assume you meant that you should add 2? If yes, that's such a mind boggling basic thing to do that I agree with you, and it makes no sense that you're being crucified.

it makes no sense that you're being crucified. Probably because there's significant overlap in the Venn diagram of people with years experience who professionally develop products that generate $millions in wealth/value, and people who would fail that interview. Or we have worked with junior developers who have really grown and flourished under our care, who would never have gotten that chance with such insane Dracon…

I don't get it. This is an extremely basic fact that most people can figure out after thinking about primes for a minute. Maybe if you ask for "what's an easy optimisation here?" This would make the candidate think more closely about invariants that their code should hold, which in itself is a very valuable skill.

Re: Code Llama, a state-of-the-art large language model for coding

#462

Post by ttyyzz »

irrational wrote: post #396
ttyyzz wrote: post #384

Earlier quoted context omitted.

It's basically the same thing in our company, too. They basically put a similar rule in place that prevents anyone from using e.g. Chat GPT. Little do they know that all software devs within the company are using co-pilot and the company is even paying for it. It's quite a funny situation tbh..

Except, at my company they block software like that. Not only do they block it, but if you try to go to it a security person will immediately call your manager and ask what you are doing.

And people can access those systems via their own devices without the company knowing.

Re: Code Llama, a state-of-the-art large language model for coding

#463

Post by RossBencina »

FrozenSynapse wrote: post #458
daemonologist wrote: post #236

Works nearly out of the box with llama.cpp, which makes it easy to try locally: https://github.com/ggerganov/llama.cpp/issues/2766 Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt): # prints the first ten prime numbers def print_primes(): i = 2 num_printed = 0 # end of prompt while num_printed It will be interesting to see how the larger models perform, especially afte…

how much VRAM do you need to run quantised 7b model?

Rough calculation: typical quantization is 4 bit, so 7b weights fit in in 3.6GB, then my rule of thumb would be 2GB for the activations and attention cache (not usually quantized). So 6 or 8 GB VRAM would probably do it. llama.cpp will let you offload your choice of layers to GPU, so you could probably get quite a way with 4GB.

Re: Code Llama, a state-of-the-art large language model for coding

#464

Post by woadwarrior01 »

selfhoster11 wrote: post #450
ohyes wrote: post #121

Earlier quoted context omitted.

What hardware do you have that lets you run 7b and do other stuff at the same time?

A 7B model at 8-bit quantization takes up 7 GB of RAM. Less if you use a 6-bit quantization, which is nearly as good. Otherwise it's just a question of having enough system RAM and CPU cores, plus maybe a small discrete GPU.

You’ll need a bit more than 7GB (~1 GB or so), even at 8 bit quantization, because of the KV-cache. LLM inference is notoriously inefficient without it, because it’s autoregressive.

Re: Code Llama, a state-of-the-art large language model for coding

#465

Post by NhanH »

maleldil wrote: post #461
warent wrote: post #405

Earlier quoted context omitted.

it makes no sense that you're being crucified. Probably because there's significant overlap in the Venn diagram of people with years experience who professionally develop products that generate $millions in wealth/value, and people who would fail that interview. Or we have worked with junior developers who have really grown and flourished under our care, who would never have gotten that chance with such insane Dracon…

I don't get it. This is an extremely basic fact that most people can figure out after thinking about primes for a minute. Maybe if you ask for "what's an easy optimisation here?" This would make the candidate think more closely about invariants that their code should hold, which in itself is a very valuable skill.

Because I know enough not to write prime testing code that resemble anything like that loop to have to care about reducing its search space. If you actually want to test my knowledge about prime number, you can ask and I will tell you about using some probabilistic choice instead, and that I know fast deterministic one might exist, but I am not up to date on the state of the art.

If I have to write the loop above, I am assuming it is the Fizzbuzz equivalent of your company to show that I know how to write a while loop. I am not thinking about reducing the search space because I am writing the code semi-unconscious and frankly just want to get to the next question.

Re: Code Llama, a state-of-the-art large language model for coding

#466

Post by RossBencina »

epolanski wrote: post #204
liuliu wrote: post #174

Earlier quoted context omitted.

34B should be able to run on 24GiB consumer graphics card, or 32GiB Mac (M1 / M2 chips) with quantization (5~6bit) (and 7B should be able to run on your smart toaster).

Are there cloud offerings to run those models on somebody's else computer? Any "eli5" tutorial on how to do so, if so? I want to give these models a run but I have no powerful GPU to run them on so don't know where to start.

runpod, togethercomputer, replicate.

Matthew Berman has a tutorial on YT showing how to use TheBloke's docker containers on runpod. Sam Witteveen has done videos on together and replicate, they both offer cloud-hosted LLM inference as a service.

Re: Code Llama, a state-of-the-art large language model for coding

#468

Post by codercowmoo »

jmiskovic wrote: post #158
esperent wrote: post #27

Earlier quoted context omitted.

I think this is called "mixture of experts" and also there's a lot of speculation that it's how GPT-4 works, although probably with just a few large models rather than many small ones.

It's been confirmed by multiple (unofficial) sources that GPT-4 is 8 models, each 220B parameters. Another rumor is GPT-4 being 16x111B models. There's a quite fresh and active project replicating something similar with herd of llamas: https://github.com/jondurbin/airoboros

Can you provide some of these sources?

Re: Code Llama, a state-of-the-art large language model for coding

#469

Post by croes »

maleldil wrote: post #333
blibble wrote: post #275

Earlier quoted context omitted.

I'd fail an interview candidate that suggested adding 1 each time for subsequent prime testing

I assume you meant that you should add 2? If yes, that's such a mind boggling basic thing to do that I agree with you, and it makes no sense that you're being crucified.

You would fail someone about such a trifle?

Re: Code Llama, a state-of-the-art large language model for coding

#470

Post by 5Qn8mNbc2FNCiVV »

squeaky-clean wrote: post #443
blibble wrote: post #275

Earlier quoted context omitted.

I'd fail an interview candidate that suggested adding 1 each time for subsequent prime testing

You'd reject a candidate that is willing and legally able to work for free while also cloning themselves so they can pair program with every one of your employees at once?

I would not ship code just because of those reasons.
Post reply on HN