Open source AI is the path forward • Hacker News Board

Re: Open source AI is the path forward

#151

Post by diggan » Tue, Jul 23, 2024, 4:26 PM UTC

valine wrote: post #135

diggan wrote: post #104

> Today we’re taking the next steps towards open source AI becoming the industry standard. We’re releasing Llama 3.1 405B, the first frontier-level open source AI model, Why do people keep mislabeling this as Open Source? The whole point of calling something Open Source is that the "magic sauce" of how to build something is publicly available, so I could built it myself if I have the means. But without the training d…

The codebase to do the training is way less valuable than the weights for the vast majority of people. Releasing the training code would be nice, but it doesn't really help anyone but Meta's direct competitors. If you want to train on top of Llama there's absolutely nothing stopping you. Plenty of open source tools to do parameter optimization.

Not just the training code but the training data as well, should be under a permissive license, otherwise you cannot call the project itself Open Source, which Facebook does here.

> is way less valuable than the weights for the vast majority of people

The same is true for most Open Source projects, most people use the distributed binaries or other artifacts from the projects, and couldn't care less about the code itself. But that doesn't warrant us changing the meaning of Open Source just because companies feel like it's free PR.

> If you want to train on top of Llama there's absolutely nothing stopping you.

Sure, but in order for the intent of Open Source to be true for Llama, I should be able to build this project from scratch. Say I have a farm of 100 A100's, could I reproduce the Llama model from scratch today?

Re: Open source AI is the path forward

#152

Post by vngzs » Tue, Jul 23, 2024, 4:27 PM UTC

diggan wrote: post #104

> Today we’re taking the next steps towards open source AI becoming the industry standard. We’re releasing Llama 3.1 405B, the first frontier-level open source AI model, Why do people keep mislabeling this as Open Source? The whole point of calling something Open Source is that the "magic sauce" of how to build something is publicly available, so I could built it myself if I have the means. But without the training d…

Agreed. The Linux kernel source contains everything you need to produce Linux kernel binaries. The llama source does not contain what you need to produce llama models. Facebook is using sleight of hand to garner favor with open model weights.

Open model weights are still commendable, but it's a far cry from open-source (or even libre) software!

Re: Open source AI is the path forward

#153

Post by bilsbie » Tue, Jul 23, 2024, 4:27 PM UTC

the8thbit wrote: post #75

bilsbie wrote: post #26

Earlier quoted context omitted.

Can’t you do fine tuning on those binaries? That’s a modification.

You can fine tune the models, and you can modify binaries. However, there is no human readable "source" to open in either case. The act of "fine tuning" is essentially brute forcing the system to gradually alter the weights such that loss is reduced against a new training set. This limits what you can actually do with the model vs an actual open source system where you can understand how the system is working and mod…

You make a good point but those are also just limitations of the technology (or at least our current understanding of it)

Maybe an analogy would help. A family spent generations breeding the perfect apple tree and they decided to “open source” it. What would open sourcing look like?

Re: Open source AI is the path forward

#154

Post by GaggiX » Tue, Jul 23, 2024, 4:27 PM UTC

Llama 3.1 405B is on par with GPT-4o and Claude 3.5 Sonnet, the 70B model is better than GPT 3.5 turbo, incredible.

Re: Open source AI is the path forward

#155

Post by itissid » Tue, Jul 23, 2024, 4:28 PM UTC

How are smaller models distilled from large models, I know of LoRA, quantization like technique; but does distilling also mean generating new datasets for conversing with smaller models entirely from the big models for many simpler tasks?

Re: Open source AI is the path forward

#156

Post by Nesco » Tue, Jul 23, 2024, 4:28 PM UTC

enriquto wrote: post #142

It's alarming that he refers to llama as if it was open source. The definition of free software (and open source, for that mater), is well-established. The same definition applies to all programs, whether they are "AI" or not. In any case, if a program was built by training against a dataset, the whole dataset is part of the source code. Llama is distributed in binary form, and it was built based on a secret dataset.…

The training data contains most likely insane amounts of copyrighted material. That’s why virtually none of the “open models” come with their training data

Re: Open source AI is the path forward

#157

Post by elromulous » Tue, Jul 23, 2024, 4:30 PM UTC

diggan wrote: post #104

> Today we’re taking the next steps towards open source AI becoming the industry standard. We’re releasing Llama 3.1 405B, the first frontier-level open source AI model, Why do people keep mislabeling this as Open Source? The whole point of calling something Open Source is that the "magic sauce" of how to build something is publicly available, so I could built it myself if I have the means. But without the training d…

100%. With this licensing model, meta gets to reap the benefits of open source (people contributing, social cachet), without any of the real detriment (exposing secret sauce).

Re: Open source AI is the path forward

#158

Post by kjkjadksj » Tue, Jul 23, 2024, 4:31 PM UTC

JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

The size of the cluster would have to be massive or else your job will be on the queue for a year. And even then what are you going to do downsize the resources requested so you can get in earlier? After a certain point it starts to make more sense to just buy your own xeons and run your own cluster.

Re: Open source AI is the path forward

#159

Post by unethical_ban » Tue, Jul 23, 2024, 4:31 PM UTC

JumpCrisscross wrote: post #113

diggan wrote: post #107

Earlier quoted context omitted.

So because it's really hard to do proper Open Source with these LLMs, means we need to change the meaning of Open Source so it fits with these PR releases?

> because it's really hard to do proper Open Source with these LLMs, means we need to change the meaning of Open Source so it fits with these PR releases? Open training data is hard to the point of impracticality. It requires excluding private and proprietary data. Meanwhile, the term "open source" is massively popular. So it will get used. The question is how. Meta et al would love for the choice to be between, on o…

>Meanwhile, the term "open source" is massively popular. So it will get used. The question is how.

Here's the source of the disagreement. You're justifying the use of the term "open source" by saying it's logical for Meta to want to use it for its popularity and layman (incorrect) understanding.

Other person is saying it doesn't matter how convenient it is or how much Meta wants to use it, that the term "open source" is misleading for a product where the "source" is the training data, and the final product has onerous restrictions on use.

This would be like Adobe giving Photoshop away for free, but for personal use only and not for making ads for Adobe's competitors. Sure, Adobe likes it and most users may be fine with it, but it isn't open source.

>The way open source software, today, generally means source available, not FOSS.

I don't agree with that. When a company says "open source" but it's not free, the tech community is quick to call it "source available" or "open core".

Re: Open source AI is the path forward

#160

Post by swyx » Tue, Jul 23, 2024, 4:31 PM UTC

cs702 wrote: post #74

> We’re releasing Llama 3.1 405B, the first frontier-level open source AI model, as well as new and improved Llama 3.1 70B and 8B models. Bravo! While I don't agree with Zuck's views and actions on many fronts, on this occasion I think he and the AI folks at Meta deserve our praise and gratitude. With this release, they have brought the cost of pretraining a frontier 400B+ parameter model to ZERO for pretty much ever…

> the AI folks at Meta deserve our praise and gratitude

We interviewed Thomas who led Llama 2 and 3 post training here in case you want to hear from someone closer to the ground on the models https://www.latent.space/p/llama-3