Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

Post reply on HN
Print view View original

221–230 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#221

Post by claytongrassick »

lordnacho wrote: post #128

Copilot has been working great for me thus far, but it's limited by its interface. It seems like it only knows how to make predictions for the next bit of text. Is anyone working on a code AI that can suggest refactorings? "You should pull these lines into a function, it's repetitive" "You should change this structure so it is easier to use" Etc

I've been using Cursor (https://www.cursor.so/) and it can do embeddings of the entire codebase, refactoring entire classes, etc. I had it rewrite a UI to add state to show one item at a time and have a selection list to the left and it executed it perfectly in MUI controls, first try.

Re: Code Llama, a state-of-the-art large language model for coding

#222

Post by redox99 »

epolanski wrote: post #204
liuliu wrote: post #174

Earlier quoted context omitted.

34B should be able to run on 24GiB consumer graphics card, or 32GiB Mac (M1 / M2 chips) with quantization (5~6bit) (and 7B should be able to run on your smart toaster).

Are there cloud offerings to run those models on somebody's else computer? Any "eli5" tutorial on how to do so, if so? I want to give these models a run but I have no powerful GPU to run them on so don't know where to start.

On runpod there is a TheBloke template with everything set up for you. An A6000 is good enough to run 70b 4bit.

Re: Code Llama, a state-of-the-art large language model for coding

#223

Post by mchiang »

syntaxing wrote: post #185

TheBloke doesn’t joke around [1]. I’m guessing we’ll have the quantized ones by the end of the day. I’m super excited to use the 34B Python 4 bit quantized one that should just fit on a 3090. [1] https://huggingface.co/TheBloke/CodeLlama-13B-Python-fp16

Ollama supports it already:

`ollama run codellama:7b-instruct`

https://ollama.ai/blog/run-code-llama-locally

More models uploaded as we speak:

https://ollama.ai/library/codellama

Re: Code Llama, a state-of-the-art large language model for coding

#224

Post by thewataccount »

msp26 wrote: post #157
thewataccount wrote: post #124

Earlier quoted context omitted.

For in-editor like copilot you can try this locally - https://github.com/smallcloudai/refact This works well for me except the 15B+ don't run fast enough on a 4090 - hopefully exllama supports non-llama models, or maybe it'll support CodeLLaMa already I'm not sure. For general chat testing/usage this works pretty well with lots of options - https://github.com/oobabooga/text-generation-webui/

>This works well for me except the 15B+ don't run fast enough on a 4090 I assume quantized models will run a lot better. TheBloke already seems like he's on it. https://huggingface.co/TheBloke/CodeLlama-13B-fp16

Unfortunately what I tested was StarCoder 4bit. We really need exllama which should make even 30b viable from what I can tell.

Because codellama is llama based it may just work possibly?

Re: Code Llama, a state-of-the-art large language model for coding

#225

Post by lsmeducation »

ilaksh wrote: post #209
lsmeducation wrote: post #202

Earlier quoted context omitted.

Continuing your aside… Humans don’t become obsolete, we become bored. This tech will make us bored. When humans get too bored and need shit to stir up, we’ll start a war. Take US and China, global prosperity is not enough right? We need to stoke the flames of war over Taiwan. In the next 300 years we’ll wipe out most of each other in some ridiculous war, and then rebuild.

I agree that WWIII is a concern but I don't think it will be brought about by boredom. "Global prosperity" might be true in a very long-term historical sense, but it's misleading to apply it to the immediate situation. Taiwan is not just a talking point. Control over Taiwan is critical for maintaining hegemony. When that is no longer assured, there will likely be a bloody battle before China is given the free reign t…

I’ll go with the 30 year mark. Countries like Russia or China don’t get humbled in a loss (like Germany didn’t in WW1). Russia will negotiate some terms for Ukraine (or maintain perpetual war), but I believe it will become a military state that will funnel all money into the defense sector. The same with Iran, and the same with China.

Iran supplies Russia with drones. I can promise you Russia will help Iran enrich their uranium. They are both pariah states, what do they have to lose? Nuclear Iran, here enters Israel.

Everyone’s arming up, there’s a gun fight coming.

Re: Code Llama, a state-of-the-art large language model for coding

#226

Post by [deleted] »

mchiang wrote: post #223
syntaxing wrote: post #185

TheBloke doesn’t joke around [1]. I’m guessing we’ll have the quantized ones by the end of the day. I’m super excited to use the 34B Python 4 bit quantized one that should just fit on a 3090. [1] https://huggingface.co/TheBloke/CodeLlama-13B-Python-fp16

Ollama supports it already: `ollama run codellama:7b-instruct` https://ollama.ai/blog/run-code-llama-locally More models uploaded as we speak: https://ollama.ai/library/codellama

[deleted]

Re: Code Llama, a state-of-the-art large language model for coding

#227

Post by littlestymaar »

riku_iki wrote: post #215
redox99 wrote: post #38

The highlight IMO > The Code Llama models provide stable generations with up to 100,000 tokens of context. All models are trained on sequences of 16,000 tokens and show improvements on inputs with up to 100,000 tokens. Edit: Reading the paper, key retrieval accuracy really deteriorates after 16k tokens, so it remains to be seen how useful the 100k context is.

> The Code Llama models provide stable generations with up to 100,000 tokens of context. what is the trick to achieve 100k context? They can't just use 100k wide transformer layer, it is cost prohibitive, right?..

I'm pretty sure they don't do that, but for code the relevant relationship between two tokens is easy to determine with the semantics of the language alone (for instance you can say that tokens related to a local variable have no relationship with tokens outside), so it would lead to a sparse matrix in the transformer, reducing the cost of big contexts by a lot. But it would require language specific preprocessing, and whether you can make it fast is also dubious. I don't think it's been tried so far.

Re: Code Llama, a state-of-the-art large language model for coding

#228

Post by criley2 »

rvnx wrote: post #216
criley2 wrote: post #159

Earlier quoted context omitted.

I don't get comments like this, we can all go and test Bard and see that what you're saying isn't true https://g.co/bard/share/95761dd6d45e

Well look for yourself: https://g.co/bard/share/e8d14854ccab The rm answer is now "hardcoded" (aka, manually entered by reviewers), the same with the prime or fibonnaci. This is why we both see the same code across different accounts (you can make the test if you are curious).

Okay, so the entire point of the comment is "A current model which does well used to be bad!"

With all due respect, is that a valuable thing to say? Isn't it true of them all?

Re: Code Llama, a state-of-the-art large language model for coding

#229

Post by syntaxing »

mchiang wrote: post #223
syntaxing wrote: post #185

TheBloke doesn’t joke around [1]. I’m guessing we’ll have the quantized ones by the end of the day. I’m super excited to use the 34B Python 4 bit quantized one that should just fit on a 3090. [1] https://huggingface.co/TheBloke/CodeLlama-13B-Python-fp16

Ollama supports it already: `ollama run codellama:7b-instruct` https://ollama.ai/blog/run-code-llama-locally More models uploaded as we speak: https://ollama.ai/library/codellama

Whoa, it’s absolutely astounding how fast the community is reacting to these model release!

Re: Code Llama, a state-of-the-art large language model for coding

#230

Post by thewataccount »

mymac wrote: post #122

Never before in the history of mankind was a group so absolutely besotted with the idea of putting themselves out of a job.

Is automation not what every engineer strives for when possible? Especially software developers.

From my experience with github copilot and GPT4 - developers are NOT going anywhere anytime soon. You'll certainly be faster though.

Post reply on HN