Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

91–100 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#91
post #7

Does anyone have a good explanation for Meta's strategy with AI? The only thing I've been able to think is they're trying to commoditize this new category before Microsoft and Google can lock it in, but where to from there? Is it just to block the others from a new revenue source, or do they have a longer game they're playing?

Clearly the research team at Meta knows the domain as well anybody, has access to a data trove as large as anybody and their distribution capability is as large scale as anyone's.

If their choice right now is not to try to overtly monetize these capabilities but instead commoditize and "democratize" what others are offering it suggests they think that a proprietary monetization route is not available to them. In other words they do not leave money on the table. They think that (at least right now) there is no money on the table that they can get to.

Rather than remaining quiet and isolated, the best alternative - their conjectured thinking goes - is to show up as they do, buying up good will with various stakeholders, maintaining mindshare internally and externally etc.

Assuming that the above reading is correct it still leaves various options as to why they may have come to that conclusion: For example reasoning about the future of this sector they might be thinking that there is no real technical moat and they simply accelerate that reality to gain some brownie points.

It may be also idiosyncratic reasons specific to their own business model (data privacy challenges and how any AI monetization will mesh with all that). The drawback of being the elephant in the room is that there is not much room to move.

The nature of their long game depends on which of the decision branches carries more weight. Maybe it is wait-and-see until others clear up the regulatory hurdles. Or keep the engines running until the real and irreducible added value of LLM algos and the like becomes clear.

Re: Code Llama, a state-of-the-art large language model for coding

#93
post #33

Earlier quoted context omitted.

You've already downloaded and thoroughly tested the 7B parameter model of "code llama"? I'm skeptical.

Likely meta employee?

I've been using this or something similar internally for months and love it. The thing that gets downright spooky is the comments believe it or not. I'll have some method with a short variable name in a larger program and not only does it often suggest a pretty good snippet of code the comments will be correct and explain what the intent behind the code is. It's just a LLM but you really start to get the feeling the whole is greater than the sum of the parts.

Re: Code Llama, a state-of-the-art large language model for coding

#96
post #38

The highlight IMO > The Code Llama models provide stable generations with up to 100,000 tokens of context. All models are trained on sequences of 16,000 tokens and show improvements on inputs with up to 100,000 tokens. Edit: Reading the paper, key retrieval accuracy really deteriorates after 16k tokens, so it remains to be seen how useful the 100k context is.

Did Meta add scalable rope to the official implementation?

We changed RoPE's theta from 10k to 1m and fine-tuned with 16k tokens long sequences.

Re: Code Llama, a state-of-the-art large language model for coding

#98
Amazing! It's great that Meta is making AI progress.

In the meantime, we are still waiting for Google to show what they have (according to their research papers, they are beating others).

> User: Write a loop in Python that displays the top 10 prime numbers.

> Bard: Sorry I am just an AI, I can't help you with coding.

> User: How to ask confirmation before deleting a file ?

> Bard: To ask confirmation before deleting a file, just add -f to the rm command.

(real cases)

Re: Code Llama, a state-of-the-art large language model for coding

#99

>The Code Llama models provide stable generations with up to 100,000 tokens of context. Not a bad context window, but makes me wonder how embedded code models would pick that context when dealing with a codebase larger than 100K tokens. And this makes me further wonder if, when coding with such a tool (or at least a knowledge that they’re becoming more widely used and leaned on), are there some new considerations tha…

I’ve found the utility of the coding LLMs gets a lot higher when you’ve got code comments and descriptive variable and function names - the LLM makes better inferences and suggestions. We’ve seen similar on data - properly tagged data and descriptive field names helps the LLM to produce much more useful responses. I’m secretly hoping the spread of these tools will finally lead my fellow developers to comment their code and stop using three character variable names.

Re: Code Llama, a state-of-the-art large language model for coding

#100
post #7

Does anyone have a good explanation for Meta's strategy with AI? The only thing I've been able to think is they're trying to commoditize this new category before Microsoft and Google can lock it in, but where to from there? Is it just to block the others from a new revenue source, or do they have a longer game they're playing?

I watched a good talk from Yann LeCun who is Chief AI Scientist at Meta, and he explained that the thinking is that open source AI models will be the long-term winner, so it's best for them to work in that arena.

https://www.youtube.com/watch?v=vyqXLJsmsrk

Post reply on HN