Open Source AI needs to include training data.
Open source AI is the path forward
201–210 of 936 posts
Re: Open source AI is the path forward
#202Earlier quoted context omitted.
the training data is the source.
> the training data is the source Sure. But that's not going to be released. The term open source AI cannot be expected to cover it because it's not practical.
Meta can call it something else other than open source.
Synthetic part of the training data could be released.
Re: Open source AI is the path forward
#203> Developers can run inference on Llama 3.1 405B on their own infra at roughly 50% the cost of using closed models like GPT-4o Does anyone have details on exactly what this means or where/how this metric gets derived?
I am guessing these are prices on services like AWS Bedrock (their post is down right now).
Re: Open source AI is the path forward
#204It's alarming that he refers to llama as if it was open source. The definition of free software (and open source, for that mater), is well-established. The same definition applies to all programs, whether they are "AI" or not. In any case, if a program was built by training against a dataset, the whole dataset is part of the source code. Llama is distributed in binary form, and it was built based on a secret dataset.…
[deleted]
Re: Open source AI is the path forward
#205Earlier quoted context omitted.
> no one has bothered to release their training data If the FOSS community sets this as the benchmark for open source in respect of AI, they're going to lose control of the term. In most jurisdictions it would be illegal for the likes of Meta to release training data.
the training data is the source.
I don’t think it’s that simple. The source is “the preferred form of the work for making modifications to it” (to use the GPL’s wording).
For an LLM, that’s not the training data. That’s the model itself. You don’t make changes to an LLM by going back to the training data and making changes to it, then re-running the training. You update the model itself with more training data.
You can’t even use the training code and original training data to reproduce the existing model. A lot of it is non-deterministic, so you’ll get different results each time anyway.
Another complication is that the object code for normal software is a clear derivative work of the source code. It’s a direct translation from one form to another. This isn’t the case with LLMs and their training data. The models learn from it, but they aren’t simply an alternative form of it. I don’t think you can describe an LLM as a derivative work of its training data. It learns from it, it isn’t a copy of it. This is mostly the reason why distributing training data is infeasible – the model’s creator may not have the license to do so.
Would it be extremely useful to have the original training data? Definitely. Is distributing it the same as distributing source code for normal software? I don’t think so.
I think new terminology is needed for open AI models. We can’t simply re-use what works for human-editable code because it’s a fundamentally different type of thing with different technical and legal constraints.
Re: Open source AI is the path forward
#206Earlier quoted context omitted.
> because it's really hard to do proper Open Source with these LLMs, means we need to change the meaning of Open Source so it fits with these PR releases? Open training data is hard to the point of impracticality. It requires excluding private and proprietary data. Meanwhile, the term "open source" is massively popular. So it will get used. The question is how. Meta et al would love for the choice to be between, on o…
>Meanwhile, the term "open source" is massively popular. So it will get used. The question is how. Here's the source of the disagreement. You're justifying the use of the term "open source" by saying it's logical for Meta to want to use it for its popularity and layman (incorrect) understanding. Other person is saying it doesn't matter how convenient it is or how much Meta wants to use it, that the term "open source"…
> You're justifying the use of the term "open source" by saying it's logical for Meta to want to use it for its popularity and layman (incorrect) understanding
I'm actually not a fan of Meta's definition. I'm arguing specifically against an unrealistic definition, because for practical purposes that cedes the term to Meta.
> the term "open source" is misleading for a product where the "source" is the training data, and the final product has onerous restrictions on use
Agree. I think the focus should be on the use restrictions.
> When a company says "open source" but it's not free, the tech community is quick to call it "source available" or "open core"
This isn't consistently applied. It's why we have the free vs open vs FOSS fracture.
Re: Open source AI is the path forward
#207Looks like it's easy to test out these models now on Together AI - https://api.together.ai
[deleted]
Re: Open source AI is the path forward
#208Just added Llama 3.1 405B/70B/8B to https://double.bot (VSCode coding assistant) if anyone would like to try it. --- Some observations: * The model is much better at trajectory correcting and putting out a chain of tangential thoughts than other frontier models like Sonnet or GPT-4o. Usually, these models are limited to outputting "one thought", no matter how verbose that thought might be. * I remember in Dec of 2022…
[deleted]
Re: Open source AI is the path forward
#209> Today we’re taking the next steps towards open source AI becoming the industry standard. We’re releasing Llama 3.1 405B, the first frontier-level open source AI model, Why do people keep mislabeling this as Open Source? The whole point of calling something Open Source is that the "magic sauce" of how to build something is publicly available, so I could built it myself if I have the means. But without the training d…
> Why do people keep mislabeling this as Open Source? The whole point of calling something Open Source is that the "magic sauce" of how to build something is publicly available, so I could built it myself if I have the means. But without the training data publicly available, could I train Llama 3.1 if I had the means? I don't think not releasing the commit history of a project makes it not Open Source, this seems lik…
For the freedom to change to be effective, a user must be given the software in a form they can modify. Can you tweak an LLM once it's built? (I genuinely don't know the answer)
Re: Open source AI is the path forward
#210[flagged]