so that North Korea will create small call centers for cheaper, since they can get these models for free?
This argument implies that cheap phones are bad since telemarketers can use them.
Open source AI is the path forward
221–230 of 936 posts
Re: Open source AI is the path forward
#222Earlier quoted context omitted.
> terms that have specific, widely accepted definitions OSF's definition is far from the only one [1]. Switzerland is currently implementing CH Open's definition, the EU another one, et cetera . > I don't get why you think it would be useful to call LLMs with published weights "open source" I don't. I'm saying that if the choice is between open weights or open weights + open training data, open weights will win becau…
For the EU, I'm guessing you're talking about the EUPL, which is FSF/OSI approved and GPL compatible, generally considered copyleft. For the CH Open, I'm not finding anything specific, even from Swiss websites, could you help me understand what you're referring to here? I'm guessing that all these definitions have at least some points in common, which involves (another guess) at least being able to produce the output…
> For the CH Open, I'm not finding anything specific, even from Swiss websites, could you help me understand what you're referring to here
Was on the HN front page earlier [1][2]. The definition comes strikingly close to source on request with no use restrictions.
> all these definitions have at least some points in common
Agreed. But they're all different. There isn't an accepted defintiion of open source even when it comes to software; there is an accepted set of broad principles.
[1] https://news.ycombinator.com/item?id=41047172
[2] https://joinup.ec.europa.eu/collection/open-source-observato...
Re: Open source AI is the path forward
#223Earlier quoted context omitted.
> In any case, if a program was built by training against a dataset, the whole dataset is part of the source code. I'm not sure why I keep seeing this. What is the equivalent of the training data for something like the Linux kernel?
> What is the equivalent of the training data for something like the Linux kernel? It's the source code. For the linux kernel: compile(sourcecode) = binary For llama: train(data) = weights
That analogy doesn't work. `train` is not a deterministic process. Meta has all of the training data and all of the supporting source code and they still won't get the same `weights` if they re-run the process.
The weights are the result of the development process, like the source code of a program is the result of a development process.
Re: Open source AI is the path forward
#224Earlier quoted context omitted.
The training data contains most likely insane amounts of copyrighted material. That’s why virtually none of the “open models” come with their training data
> The training data contains most likely insane amounts of copyrighted material. If that is the case then the weights must inherit all these copyrights. It has been shown (at least in image processing) that you can extract many training images from the weights, almost verbatim. Hiding the training data does not solve this issue. But regardless of copyright issues, people here are complaining about the malicious use o…
> If that is the case then the weights must inherit all these copyrights.
Not if it's a fair use (which is obviously the defence they're hoping for)
Re: Open source AI is the path forward
#225Small language models is the path forward
https://medium.com/thoughts-on-machine-learning/small-langua...
Re: Open source AI is the path forward
#226How are smaller models distilled from large models, I know of LoRA, quantization like technique; but does distilling also mean generating new datasets for conversing with smaller models entirely from the big models for many simpler tasks?
Smaller models can be trained to match log probs of the larger model.
Larger model can be used to generate synthethic data for the smaller model.
Re: Open source AI is the path forward
#227Note that Meta's models are not open source in any interpretation of the term. * You can't use them for any purpose. For example, the license prohibits using these models to train other models. * You can't meaningfully modify them given there is almost no information available about the training data, how they were trained, or how the training data was processed. As such, the model itself is not available under an op…
> you can't meaningfully modify them given there is almost no information available about the training data, how they were trained, or how the training data was processed. I was under the impression that you could still fine-tune the models or apply your own RLHF on top of them. My understanding is that the training data would mostly be useful for training the model yourself from scratch (possibly after modifying the…
From what i understand the training data and careful curation of it is the hard part. Everyone wants training data sets to train their own models instead of producing their own.
Re: Open source AI is the path forward
#228Huge companies like facebook will often argue for solutions that on the surface, seem to be in the public interest. But I have strong doubts they (or any other company) actually believe what they are saying. Here is the reality: - Facebook is spending untold billions on GPU hardware. - Facebook is arguing in favor of open sourcing the models, that they spent billions of dollars to generate, for free...? It follows th…
Meta is, fundamentally, a user-generated-content distribution company. Meta wants to make sure they commoditize their complements: they don’t want a world where OpenAI captures all the value of content generation, they want the cost of producing the best content to be as close to free as possible.
[deleted]
Re: Open source AI is the path forward
#229Cynically I think this position is largely due to how they can undercut OpenAI's moat.
It's not cynical, it's just an awareness that public companies have a fiduciary duty to their shareholders.
Re: Open source AI is the path forward
#230“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…
How about using some of that money to develop CUDA alternatives so everyone is not paying the Nvidia tax?
Either you port Tensorflow (Apple)[1] or PyTorch to your platform or you allow CUDA to run on your hardware (AMD) [2]. Companies are incentives to not have NVIDIA having a monopoly but the thing is that CUDA is a huge moat due to compatibility of all frameworks and everyone knows it. Also, all of the cloud or on premises providers use NVIDIA regardless.
[1] https://developer.apple.com/metal/tensorflow-plugin/ [2] https://www.xda-developers.com/nvidia-cuda-amd-zluda/