Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

Post reply on HN
Print view View original

161–170 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#161

Post by rvz »

lolinder wrote: post #7

Does anyone have a good explanation for Meta's strategy with AI? The only thing I've been able to think is they're trying to commoditize this new category before Microsoft and Google can lock it in, but where to from there? Is it just to block the others from a new revenue source, or do they have a longer game they're playing?

> Does anyone have a good explanation for Meta's strategy with AI?

Yes. I said it many times. Meta is already at the finish line in the AI race to zero. All the other cloud-based AI models cannot increase their prices given that a $0 free AI model is available to be self-hosted or used on-device for private / compliance reasons.

Cloud-based AI models cannot afford to compete with free or close to free. It costs Meta close to nothing to release a readily available $0 AI model which is good enough for most use-cases that ChatGPT has already done.

> The only thing I've been able to think is they're trying to commoditize this new category before Microsoft and Google can lock it in, but where to from there? Is it just to block the others from a new revenue source, or do they have a longer game they're playing?

Mostly benefits the PyTorch ecosystem which Meta has an active community around it.

Re: Code Llama, a state-of-the-art large language model for coding

#162

Post by Palmik »

The best model, Unnatural Code Llama, is not released. Likely because it's trained on GPT4 based data, and might violate OpenAI TOS, because as per the "Unnatural" paper [1], the "unnatural" data is generated with the help of some LLM -- and you would want to use as good of an LLM as possible.

[1] https://arxiv.org/pdf/2212.09689.pdf

Re: Code Llama, a state-of-the-art large language model for coding

#163

Post by jonchurch_ »

nabakin wrote: post #138
redox99 wrote: post #38

The highlight IMO > The Code Llama models provide stable generations with up to 100,000 tokens of context. All models are trained on sequences of 16,000 tokens and show improvements on inputs with up to 100,000 tokens. Edit: Reading the paper, key retrieval accuracy really deteriorates after 16k tokens, so it remains to be seen how useful the 100k context is.

Looks like they aren't releasing a pretty interesting model too. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model/finetune on every benchmark except for slightly losing to Code Llama Python on MBPP pass@100 and slightly losing to GPT-4 on HumanEval pass@1 which is insane. Meta says later on that they aren't releasing it and give no explanation. I wonder why given how inc…

The paper states it was instruction fine tuned with synthetic data (LLM generated instructions) ala another paper (“Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor”).

The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo.

https://github.com/orhonovich/unnatural-instructions

Re: Code Llama, a state-of-the-art large language model for coding

#165

Post by losteric »

nologic01 wrote: post #91
lolinder wrote: post #7

Does anyone have a good explanation for Meta's strategy with AI? The only thing I've been able to think is they're trying to commoditize this new category before Microsoft and Google can lock it in, but where to from there? Is it just to block the others from a new revenue source, or do they have a longer game they're playing?

Clearly the research team at Meta knows the domain as well anybody, has access to a data trove as large as anybody and their distribution capability is as large scale as anyone's. If their choice right now is not to try to overtly monetize these capabilities but instead commoditize and "democratize" what others are offering it suggests they think that a proprietary monetization route is not available to them. In othe…

Well Facebook is a walled garden, perhaps the board hopes free highly capable LLMs will continue degrading the internet outside those walls thus acting as a moat for their money printer.

Re: Code Llama, a state-of-the-art large language model for coding

#167

Post by visarga »

throwaway290 wrote: post #65
vessenes wrote: post #34

Earlier quoted context omitted.

They are behind commercially, very behind. They also don't have the same economic setup and DNA as MS/OpenAI. Large corporate customers don't pay for access to the FB cloud, nor are they likely to -- Ellison has spent years building out Oracle Cloud, and he's on the FB board, for example. And I bet you didn't think of using Oracle's Cloud for your last project. So, your company DNA is free-to-all social based on ad m…

Commercially it's not clear if there is a reliable "ahead", I'd be surprised if copyright lawsuits don't start hitting MS/OAI when publishers wake up and if you take out that training data where does it leave their models?

Countries putting copyright above AI progress will just fall behind. It's one thing to demand no exact replication of copyrighted content, another to forbid training on copyrighted works. Ideas were not supposed to be under copyright, only expression, from what I remember.

Re: Code Llama, a state-of-the-art large language model for coding

#168

Post by lsmeducation »

mymac wrote: post #132
ttul wrote: post #130

Earlier quoted context omitted.

That’s just one perspective… Another perspective is that LLMs enable programmers to skip a lot of the routine and boring aspects of coding - looking up stuff, essentially - so they can focus on the fun parts that engage creativity.

But it won't stop there. Why would it stop at some arbitrarily defined boundary? The savings associated with no longer having to pay programmers the amounts of money that they believe they are worth (high enough to result in collusion between employers) are just too tempting.

Okay, think about it this way. This thing helps generate tons and tons of code. The more code people (or this thing) writes, the more shit there is to debug. More and more code, each calling each other means more and more insane bugs.

We’re going to move from debugging some crap the last developer wrote to debugging an order of magnitude more code the last developer generated.

It’s going to be wonderful for job prospects really.

Re: Code Llama, a state-of-the-art large language model for coding

#169

Post by redox99 »

Palmik wrote: post #162

The best model, Unnatural Code Llama, is not released. Likely because it's trained on GPT4 based data, and might violate OpenAI TOS, because as per the "Unnatural" paper [1], the "unnatural" data is generated with the help of some LLM -- and you would want to use as good of an LLM as possible. [1] https://arxiv.org/pdf/2212.09689.pdf

The good thing is that if it's only finetuned on 15k instructions, we should see a community made model like that very soon.

Re: Code Llama, a state-of-the-art large language model for coding

#170

Post by roughly »

vladms wrote: post #150
roughly wrote: post #106

Earlier quoted context omitted.

That’s interesting. I tend to lump FB, Amazon, Google, and MS in my head when thinking about the tech giants, but you’re right, FB is the only one of those not offering a commercial platform. For them, building out the capabilities of the LLMs is something to be done in the open with community involvement, because they’re not going to monetize the models themselves. They’re also getting a fantastic amount of press fr…

Depends what you mean by platform and depends what you mean by FB. If by FB you mean Meta, they have also https://www.workplace.com/ (which is like an internal facebook), instagram, whatsapp and some others. Integration of LLMs technology in those "platform" might give them some advantage.

Right, but they’re not competing directly on offering the LLM - they benefit from having a better LLM as a feature, but their value add is elsewhere in the product.
Post reply on HN