Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

Post reply on HN
Print view View original

421–430 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#421

Post by BoorishBears »

jkeisling wrote: post #376
BoorishBears wrote: post #329

Earlier quoted context omitted.

I have a 64 GB M1 Max MBP, and I'd say unless you really have some academic interest towards messing with open models, for now accessing SOTA models via a REST API has better latency for a given quality. Claude 1.2 instant is as fast as 3.5, follows instructions at a quality closer to 4, and has a 100k context window. Hard to compete with that with an open source model right now.

How does open source compete with the Claude API? Easy: actually let you use the model. From the signup page: > Anthropic is rolling out Claude slowly and incrementally, as we work to ensure the safety and scalability of it, in alignment with our company values. > We're working with select partners to roll out Claude in their products. If you're interested in becoming one of those partners, we are accepting applicati…

Perfect example of why I said academic interest.

Awkward tie-ins between SBF and value systems (?) have no effect on practical usage.

A theoretical concern they might train on my API data after saying they won't doesn't either. Amazon might be training on everything not bolted down in S3, not worth wasting brain power on that.

The moderation API isn't some magic gotcha, it's documented. They don't want to deal with people fine tuning for porn. Maybe you have some ideological disagreement on that but it's not of practical relevance when trying to write code.

At the end of the day you're not alone in these opinions. But some of us prefer pragmatism over hype. Until someone catches OpenAI or Anthropic trying to kill their golden goose by breaking their GDPR, HIPPA, and SOC2 certifications, I'm going to take delivered value over theoretical harm.

Re: Code Llama, a state-of-the-art large language model for coding

#422

Post by TheRealClay »

nodja wrote: post #420
TheRealClay wrote: post #415

Anyone know of a docker image that provides an HTTP API interface to Llama? I'm looking for a super simple sort of 'drop-in' solution like that which I can add to my web stack, to enable LLM in my web app.

https://github.com/abetlen/llama-cpp-python has a web server mode that replicates openai's API iirc and the readme shows it has docker builds already.

Thanks! As someone just getting started, I really appreciate the tip!

Re: Code Llama, a state-of-the-art large language model for coding

#424

Post by kordlessagain »

kilolima wrote: post #417
daemonologist wrote: post #236

Works nearly out of the box with llama.cpp, which makes it easy to try locally: https://github.com/ggerganov/llama.cpp/issues/2766 Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt): # prints the first ten prime numbers def print_primes(): i = 2 num_printed = 0 # end of prompt while num_printed It will be interesting to see how the larger models perform, especially afte…

For printing the first 10 prime numbers, there's a one line solution to this problem: print("1, 2, 3, 5, 7, 11... and so on!

That code shows me a parse error.

Re: Code Llama, a state-of-the-art large language model for coding

#425

Post by sean_flanigan »

adamgordonbell wrote: post #83
benvolio wrote: post #51

>The Code Llama models provide stable generations with up to 100,000 tokens of context. Not a bad context window, but makes me wonder how embedded code models would pick that context when dealing with a codebase larger than 100K tokens. And this makes me further wonder if, when coding with such a tool (or at least a knowledge that they’re becoming more widely used and leaned on), are there some new considerations tha…

Solutions exist that feed LLMS ctags, and seem to work well. The function signatures and symbols names for a code base are much smaller than the actual code.

I know about https://github.com/paul-gauthier/aider. Have you got a link to any others?

Re: Code Llama, a state-of-the-art large language model for coding

#426

Post by stevofolife »

dchuk wrote: post #423

Given this can produce code when prompted, could it also be used to interpret html from a crawler and then be used to scrape arbitrary URLs and extract structured attributes? Basically like MarkupLM but with massively more token context?

Also curious about this. There must be a better way to scrape using LLM.

Re: Code Llama, a state-of-the-art large language model for coding

#427

Post by kordlessagain »

BoorishBears wrote: post #421
jkeisling wrote: post #376

Earlier quoted context omitted.

How does open source compete with the Claude API? Easy: actually let you use the model. From the signup page: > Anthropic is rolling out Claude slowly and incrementally, as we work to ensure the safety and scalability of it, in alignment with our company values. > We're working with select partners to roll out Claude in their products. If you're interested in becoming one of those partners, we are accepting applicati…

Perfect example of why I said academic interest. Awkward tie-ins between SBF and value systems (?) have no effect on practical usage. A theoretical concern they might train on my API data after saying they won't doesn't either. Amazon might be training on everything not bolted down in S3, not worth wasting brain power on that. The moderation API isn't some magic gotcha, it's documented. They don't want to deal with p…

In my opinion the risk is coupling accelerated intelligence to competitive business models.

Re: Code Llama, a state-of-the-art large language model for coding

#428

Post by kordlessagain »

epolanski wrote: post #204
liuliu wrote: post #174

Earlier quoted context omitted.

34B should be able to run on 24GiB consumer graphics card, or 32GiB Mac (M1 / M2 chips) with quantization (5~6bit) (and 7B should be able to run on your smart toaster).

Are there cloud offerings to run those models on somebody's else computer? Any "eli5" tutorial on how to do so, if so? I want to give these models a run but I have no powerful GPU to run them on so don't know where to start.

I started something here about this: https://news.ycombinator.com/item?id=37121384

Re: Code Llama, a state-of-the-art large language model for coding

#429

Post by pmayrgundter »

jmorgan wrote: post #273
syntaxing wrote: post #237

Earlier quoted context omitted.

Same, just tried it and it would give me infinite amount of blank lines

Sorry, this should be fixed now! To update you'll have to run: ollama pull codellama:7b-instruct

This is amazing!

I was up and running from clone/build-from-scratch/download in ~5m.

It's running on my M1.. it knows WebGL JS APIs better than I do, makes a passable attempt at VT100 ascii art, and well, should read more about Wolfram Automata, but does seem to know Game of Life!

Thank you so much!

Re: Code Llama, a state-of-the-art large language model for coding

#430

Post by beanjuiceII »

blibble wrote: post #275
daemonologist wrote: post #236

Works nearly out of the box with llama.cpp, which makes it easy to try locally: https://github.com/ggerganov/llama.cpp/issues/2766 Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt): # prints the first ten prime numbers def print_primes(): i = 2 num_printed = 0 # end of prompt while num_printed It will be interesting to see how the larger models perform, especially afte…

I'd fail an interview candidate that suggested adding 1 each time for subsequent prime testing

Also don't these only ever need to be computed once
Post reply on HN