Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

Post reply on HN
Print view View original

181–190 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#181

Post by outside1234 »

bryanlyon wrote: post #178

Llama is a very cool language model, it being used for coding was all but inevitable. I especially love it being released open for everyone. I do wonder about how much use it'll get, seeing as running a heavy language model on local hardware is kinda unlikely for most developers. Not everyone is runnning a system powerful enough to equip big AIs like this. I also doubt that companies are going to set up large AIs for…

... "seeing as running a heavy language model on local hardware is kinda unlikely for most developers"

for now it is :) but with quantization advances etc. it is not hard to see the trajectory.

Re: Code Llama, a state-of-the-art large language model for coding

#182

Post by ctoth »

bryanlyon wrote: post #178

Llama is a very cool language model, it being used for coding was all but inevitable. I especially love it being released open for everyone. I do wonder about how much use it'll get, seeing as running a heavy language model on local hardware is kinda unlikely for most developers. Not everyone is runnning a system powerful enough to equip big AIs like this. I also doubt that companies are going to set up large AIs for…

As we all know, computers stay the same and rarely improve.

Re: Code Llama, a state-of-the-art large language model for coding

#183

Post by manicennui »

dangerwill wrote: post #103

It's really sad how everyone here is fawning over tech that will destroy you own livelihoods. "AI won't take your job, those who use AI will" is purely short term, myopic thinking. These tools are not aimed to help workers, the end goal is to make it so you don't need to be an engineer to build software, just let the project manager or director describe the system they want and boom there it is. You can scream that t…

I'm not worried because it solves a problem for the semi-competent.

Re: Code Llama, a state-of-the-art large language model for coding

#184

Post by redox99 »

Draiken wrote: post #166

As a complete noob at actually running these models, what kind of hardware are we talking here? Couldn't pick that up from the README. I absolutely love the idea of using one of these models without having to upload my source code to a tech giant.

If you want to run them fast, a 12GB GPU (e.g 3060) for the 13B and a 24GB GPU for the 34B (e.g 3090). Otherwise llama.cpp CPU inference would work on most machines.

Re: Code Llama, a state-of-the-art large language model for coding

#185

Post by syntaxing »

TheBloke doesn’t joke around [1]. I’m guessing we’ll have the quantized ones by the end of the day. I’m super excited to use the 34B Python 4 bit quantized one that should just fit on a 3090.

[1] https://huggingface.co/TheBloke/CodeLlama-13B-Python-fp16

Re: Code Llama, a state-of-the-art large language model for coding

#186

Post by airgapstopgap »

Tostino wrote: post #31
jspisak wrote: post #13

Earlier quoted context omitted.

It would be interesting to understand if a ~30B Llama-2 model would be interesting and for what reasons.

Better reasoning and general performance than 13b by far (if llama1 was any indication), and like the other user said, can fit on a single 24gb vram gaming card, and can be peft fine-tuned with 2x 24gb cards.

Llama-1-33B was trained on 40% more tokens than LLama-1-13B; this explained some of the disparity. This time around they both have the same data scale (2T pretraining + 500B code finetune), but 34B is also using GQA which is slightly more noisy than MHA. Furthermore, there have been some weird indications in the original LLama-2 paper that 34B base model is something… even more special, it's been trained on a separate internal cluster with undervolted/underclocked GPUs (though this in itself can't hurt training results), its scores are below expectations, it's been less "aligned". Here, Code-Llama-Instruct-13B is superior to 34B on HumanEval@1. So yes, it's desirable but I wouldn't get my hopes up.

Re: Code Llama, a state-of-the-art large language model for coding

#187

Post by visarga »

benvolio wrote: post #51

>The Code Llama models provide stable generations with up to 100,000 tokens of context. Not a bad context window, but makes me wonder how embedded code models would pick that context when dealing with a codebase larger than 100K tokens. And this makes me further wonder if, when coding with such a tool (or at least a knowledge that they’re becoming more widely used and leaned on), are there some new considerations tha…

A good practice is to have a prompt file where you keep the information you want the model to have at its disposal. Then you put it in the start of your conversations with GPT-4. It's also good documentation for people.

You start a project by defining the task. Then as you iterate, you can add new information to the prompt. But it can be also partially automated - the model can have a view of the file structure, classes, routes, assets and latest errors.

I was really hoping that the one year update of Codex would be that - a LLM that can see deep into the project, not just code, but runtime execution, debugging, inspecting and monitoring. Something that can iterate like autoGPT. Unfortunately it didn't improve much and has weird conflicts with the native code completion in VSCode, you get freezes or doubled brackets.

Re: Code Llama, a state-of-the-art large language model for coding

#189

Post by idopmstuff »

lolinder wrote: post #7

Does anyone have a good explanation for Meta's strategy with AI? The only thing I've been able to think is they're trying to commoditize this new category before Microsoft and Google can lock it in, but where to from there? Is it just to block the others from a new revenue source, or do they have a longer game they're playing?

Meta has a clear channel to leverage generative AI in profitable ways in their ads. At some point in the probably not so far future, everybody's going to have custom ads generated for them that are optimized to get that particular person to click/buy/etc. Those will convert well, and the better ads convert, the more businesses will be willing to pay Meta for a given ad.

This compares favorably with Google, which is as likely to cannibalize its search business with generative AI as to create new value for itself.

Thus, for all the gen AI stuff like this, for which Meta doesn't have an obvious path to commercialization, it makes sense to release it publicly. They get plenty of benefits from this - for one, engineers (and smart people generally) who are working on really complex problems like to be able to talk about the work they're doing. If you're picking between jobs at Meta and Google, the fact that Meta's going to release your stuff publicly might well be the deciding factor.

I would also argue that there's an economic incentive. Right now, being seen as an AI company is definitely a positive for your multiple. I think the movement of Meta's stock price over the last 12 months relative to their change in profit and revenue is certainly driven in part by the perception that they're a leader in AI.

Re: Code Llama, a state-of-the-art large language model for coding

#190

Post by lsmeducation »

dangerwill wrote: post #103

It's really sad how everyone here is fawning over tech that will destroy you own livelihoods. "AI won't take your job, those who use AI will" is purely short term, myopic thinking. These tools are not aimed to help workers, the end goal is to make it so you don't need to be an engineer to build software, just let the project manager or director describe the system they want and boom there it is. You can scream that t…

It’s less of a concern if you are in mid career. But someone should warn all these college kids that are going into comp sci. I don’t think this will be the kind of lucrative field they think it’s going to be over the course of a 40 year career.

The days of getting paid well for making crud are numbered (which most of us do, even in the most interesting problem spaces).

# need a front end boilerplate that hits a backend with the following end points. REST api for movies catalogue, and a corresponding ui. Oh, unit tests please. Go with a responsive design and also make a React Native version (matter of fact provision it to my iPhone). Decide between Heroku or AWS, set up deploy with git hooks.

# scrape IMDb for initial population of the db

# I think a Reddit like comment system would be good to add, so add it. No upvote/downvote though

# handle user login with google/fb/email

# also make an admin page to manage all this

I guess the technical product designer will be the new unicorn.

Post reply on HN