Meta AI releases Code Llama 70B • Hacker News Board

Re: Meta AI releases Code Llama 70B

#91

Post by Philpax » Mon, Jan 29, 2024, 6:17 PM UTC

colesantiago wrote: post #57

fragmede wrote: post #26

Earlier quoted context omitted.

Model available , not open source. These models aren't open source because we don't have access to the data sets, nor the full code to train them, so we can't recreate the models even if we had the GPU time available to recreate them.

Everyone using AI in production is using Pytorch by Meta. Which is open source. I do not know anybody important in the AI space apart from Google using TensorFlow.

That may be true, but it's largely irrelevant. The ML framework in use has no bearing on whether or not you have the data required to reproduce the model being trained with that framework.

Re: Meta AI releases Code Llama 70B

#92

Post by simonw » Mon, Jan 29, 2024, 6:18 PM UTC

Here's the model on Hugging Face: https://huggingface.co/codellama/CodeLlama-70b-hf

Re: Meta AI releases Code Llama 70B

#93

Post by imjonse » Mon, Jan 29, 2024, 6:18 PM UTC

whimsicalism wrote: post #45

turnsout wrote: post #28

Earlier quoted context omitted.

Agreed—I hope Meta studied Deepseek's approach. The idea of a Deepseek Coder at 70B would be exciting.

The “approach” is likely just training on more tokens.

It's the quality of data and the method of training, not just the amount of tokens (per their paper release a few days ago)

Re: Meta AI releases Code Llama 70B

#94

Post by pennomi » Mon, Jan 29, 2024, 6:19 PM UTC

elorant wrote: post #68

austinpena wrote: post #5

Earlier quoted context omitted.

I'm really curious what their goal is

Prevent OpenAI from dominating the market, and at the same time have the research community enhance your models and identify key use cases.

Meta basically got a ton of free R&D that directly applies to their model architecture. Their next generation AIs will always benefit from the techniques/processes developed by the clever researchers and hobbyists out there.

Re: Meta AI releases Code Llama 70B

#95

Post by huac » Mon, Jan 29, 2024, 6:19 PM UTC

nemothekid wrote: post #82

edweis wrote: post #79

How come a company as big as Meta still uses bit.ly ?

What else would they use?

their own shortener, e.g. fb.me, presumably

Re: Meta AI releases Code Llama 70B

#96

Post by andy99 » Mon, Jan 29, 2024, 6:20 PM UTC

bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

I think a big part of it is just because they have a big AI lab. I don't know the genesis of that, but it has for years been a big contributor, see pytorch, models like SEER, as well as being one of the dominant publishers at big conferences.

Maybe now their leadership wants to push for practicality so they don't end up like Google (also a research powerhouse but failing to convert to popular advances) so they are publicly pushing strong LLMs.

Re: Meta AI releases Code Llama 70B

#97

Post by observationist » Mon, Jan 29, 2024, 6:21 PM UTC

eurekin wrote: post #75

bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

Total speculation: Yann LeCun is there and he is really passionate about the technology and openness

The faux-open models mean the models can't be used in competing products. The open code base means enthusiasts and amateurs and other people hack on Meta projects and contribute improvements.

They get free R&D and suppress competition, while looking like they have principles. Yann is clueless about open source principles, or the models would have been Apache or some other comparably open license. It's all ruthless corporate strategy, regardless of the mouth noises coming out of various meta employees.

Re: Meta AI releases Code Llama 70B

#98

Post by Lerc » Mon, Jan 29, 2024, 6:22 PM UTC

bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

If they hadn't opened the models the llama series would just be a few sub-GPT4 models. Opening the models has created a wealth of development that has built upon those models.

Alone, it was unlikely they would become a major player in a field that might be massively important. With a large community building upon their base they have a chance to influence the direction of development and possibly prevent a proprietary monopoly in the hands of another company.

Re: Meta AI releases Code Llama 70B

#99

Post by nemothekid » Mon, Jan 29, 2024, 6:22 PM UTC

3pt14159 wrote: post #87

nemothekid wrote: post #82

Earlier quoted context omitted.

What else would they use?

Something like meta.com/our_own_tech_handles_this

Not sure it's preferable to hire people at fb salaries to maintain a link shortener than just to use a reputable free one?

Re: Meta AI releases Code Llama 70B

#100

Post by datadrivenangel » Mon, Jan 29, 2024, 6:24 PM UTC

bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

AI puts pressure on search, cutting into google's ad revenue. Meta's properties are less immune to pressure from AI.