Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

Post reply on HN
Print view View original

491–500 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#491

Post by orm »

daemonologist wrote: post #236

Works nearly out of the box with llama.cpp, which makes it easy to try locally: https://github.com/ggerganov/llama.cpp/issues/2766 Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt): # prints the first ten prime numbers def print_primes(): i = 2 num_printed = 0 # end of prompt while num_printed It will be interesting to see how the larger models perform, especially afte…

How did you get access to the model?

I have been waiting for weeks, and am still waiting, to get access to Llama2 (released a month+ ago), and access to this model goes through the same form, so I'm not very hopeful. Are you getting it from other methods?

Re: Code Llama, a state-of-the-art large language model for coding

#492

Post by madduci »

orm wrote: post #491
daemonologist wrote: post #236

Works nearly out of the box with llama.cpp, which makes it easy to try locally: https://github.com/ggerganov/llama.cpp/issues/2766 Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt): # prints the first ten prime numbers def print_primes(): i = 2 num_printed = 0 # end of prompt while num_printed It will be interesting to see how the larger models perform, especially afte…

How did you get access to the model? I have been waiting for weeks, and am still waiting, to get access to Llama2 (released a month+ ago), and access to this model goes through the same form, so I'm not very hopeful. Are you getting it from other methods?

They have updated their readme in the GitHub repository

Re: Code Llama, a state-of-the-art large language model for coding

#493

Post by adamgordonbell »

sean_flanigan wrote: post #425
adamgordonbell wrote: post #83

Earlier quoted context omitted.

Solutions exist that feed LLMS ctags, and seem to work well. The function signatures and symbols names for a code base are much smaller than the actual code.

I know about https://github.com/paul-gauthier/aider . Have you got a link to any others?

Aider was exactly what I was thinking of!

Re: Code Llama, a state-of-the-art large language model for coding

#494

Post by pbhjpbhj »

bbor wrote: post #214
pbhjpbhj wrote: post #149

Earlier quoted context omitted.

The problem with 2 is that the people in power, and with immense wealth, remain there because of capitalism. They have the political power, and the resources, to enact the change ... but they also lose the most (unless you count altruism as gain, which of it were true the World would be so different). We could structure things so that LLM, and the generalised AIs to come, benefit the whole of society ... but we know…

Yes but the common man has won before! There has never been a perfect revolution/paradigm shift (personally anti utopia-through-intense-bloodshed, so hesitant to use the former term alone), but there have been many, all of which were against the wishes of those in power. Plus, if these AIs are enough to change everything, that kinda implies that we've developed flexible, reliable AGI systems. In such a world, everyth…

I like your optimism, but AGI owned by the elite to my mind will just be another opportunity to collect rent from the masses, and reduce wages.

Re: Code Llama, a state-of-the-art large language model for coding

#495

Post by Mawr »

blibble wrote: post #342
jpeterson wrote: post #325

Earlier quoted context omitted.

Having "extremely basic understanding" of prime numbers immediately at one's command is important for approximately 0% of software engineering jobs. If you instant-fail a candidate for this, it says a lot more about you and your organization than the candidate.

> If you instant-fail a candidate for this, it says a lot more about you and your organization than the candidate. yes, we expect professional software developers to have basic maths skills "what is a prime number" is taught to 7 year olds, it's not vector calculus what else would you consider to be an unreasonable thing for an employer to require? reading and writing skills of a typical 7 year old?

You happen to remember a particular piece of knowledge, so you project that expectation onto others. Theory of mind.

> yes, we expect professional software developers to have basic maths skills

Skill != knowledge. "What is a prime number" can be looked up and understood by any competent programmer in > "what is a prime number" is taught to 7 year olds, it's not vector calculus

Then it's reasonable to expect that an interviewee would be able to learn it as well, given the same resources. It does not however follow that an interviewee would inherently have that knowledge, just because 7 year olds are taught it.

Bottom line is, you're making too many assumptions about complete strangers.

Re: Code Llama, a state-of-the-art large language model for coding

#496

Post by rini17 »

Mawr wrote: post #495
blibble wrote: post #342

Earlier quoted context omitted.

> If you instant-fail a candidate for this, it says a lot more about you and your organization than the candidate. yes, we expect professional software developers to have basic maths skills "what is a prime number" is taught to 7 year olds, it's not vector calculus what else would you consider to be an unreasonable thing for an employer to require? reading and writing skills of a typical 7 year old?

You happen to remember a particular piece of knowledge, so you project that expectation onto others. Theory of mind. > yes, we expect professional software developers to have basic maths skills Skill != knowledge. "What is a prime number" can be looked up and understood by any competent programmer in > "what is a prime number" is taught to 7 year olds, it's not vector calculus Then it's reasonable to expect that an i…

If they know "prime number" is some technical term to look up. They might confuse it with amazon prime or anything else depending on context. You waste time explaining, they get indignant they are supposed to coding not do maths, complete mess.

Re: Code Llama, a state-of-the-art large language model for coding

#497

Post by worldsayshi »

ingridpan wrote: post #359
worldsayshi wrote: post #349

Earlier quoted context omitted.

Looks really promising. I wonder if the similar pricing to OpenAI means that Gradient is also(?) bleeding money even if they get a good customer base. Or are these prices sustainable over time?

Good question, esp as Gradient fine-tuning is so much cheaper than Open AI's

Yeah it's even cheaper. Although it looks like it's about the same in proportion to approx model size/expected quality? They haven't launched any >13B model yet, although they plan to.

Re: Code Llama, a state-of-the-art large language model for coding

#498

Post by beyang »

stuzenz wrote: post #353
adocomplete wrote: post #152

Earlier quoted context omitted.

Give Cody a try! (Cody.dev) With Cody you can create embeddings for your entire repo, so Cody will have much greater context about your code base and the problems you're trying to solve. Disclaimer: I just joined Sourcegraph a few weeks ago.

Cody is great, it had become my go-to (and I pay for Github Co-pilot). With that said, they have recently changed the architecture, with the local install required, and I have not managed (yet) to get it working with NixOS. Once I have some more time, I will try again - it looks like there will be some hoops to go through. https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-appimageT... Kudos to the Source Graph team,…

Hello, Sourcegrapher here! We are making some simplifications to eliminate the need for the separate local app install. Should be out in the next month or so. Until then, apologies for the added friction here, we're working hard to smooth it out.

Re: Code Llama, a state-of-the-art large language model for coding

#499

Post by selfhoster11 »

FrozenSynapse wrote: post #459
selfhoster11 wrote: post #450

Earlier quoted context omitted.

A 7B model at 8-bit quantization takes up 7 GB of RAM. Less if you use a 6-bit quantization, which is nearly as good. Otherwise it's just a question of having enough system RAM and CPU cores, plus maybe a small discrete GPU.

how's the generation speed on CPU?

On Ryzen 5600X, 7B and 13B run quite fast. Off the top of my head, pure CPU performance is about 25% slower than with an NVIDIA GPU of some kind. I don't remember the numbers off the top of my head, but the generation speed only starts to get annoying for 33B+ models.

Re: Code Llama, a state-of-the-art large language model for coding

#500

Post by selfhoster11 »

woadwarrior01 wrote: post #464
selfhoster11 wrote: post #450

Earlier quoted context omitted.

A 7B model at 8-bit quantization takes up 7 GB of RAM. Less if you use a 6-bit quantization, which is nearly as good. Otherwise it's just a question of having enough system RAM and CPU cores, plus maybe a small discrete GPU.

You’ll need a bit more than 7GB (~1 GB or so), even at 8 bit quantization, because of the KV-cache. LLM inference is notoriously inefficient without it, because it’s autoregressive.

6-bit quantizations are supposed to be nearly equivalent to 8-bit, and that does chop 1.5 GB off the model size. I think a 6-bit model should therefore fit, or if that doesn't, 5-bit medium or 5-bit small surely will.

There is always an option to go down the list of available quantizations notch by notch until you find the largest model that works. llama.cpp offers a lot of flexibility in that regard.

Post reply on HN