621–630 of 936 posts
Earlier quoted context omitted.
Where do you get access to those as a member of the general public?
I'm going to guess it's Compute Canada, which I don't think we non-academics have access to.
I wish that wasn't the case though!
Earlier quoted context omitted.
"Open weights" means you can use the weights for free (as in beer). "Open source" means you get the training dataset and the methodology. ~Nobody does open source LLMs.
Linux is open source and is mostly C code. You cannot run C code directly, you have to compile it and produce binaries. But it's the C code, not binary form, where the collaboration happens. With LLMs, weights are the binary code: it's how you run the model. But to be able to train the model from scratch, or to collaborate on new approaches, you have to operate at a the level of architecture, methods, and training da…
Earlier quoted context omitted.
> If so, then how can current ML models be open source? The source of a language model is the text it was trained on. Llama models are not open source (contrary to their claims), they are open weight.
You can find the entire Llama 3.0 pretraining set here: https://huggingface.co/datasets/HuggingFaceFW/fineweb 15T tokens, 45 terrabytes. Seems fairly open source to me.
Earlier quoted context omitted.
"Open weights" is a more appropriate term but I'll point out that these weights are also largely inscrutable to the people with the code that trained it. And for licensing reasons, the datasets may not be possible to share. There is still a lot of modifying you can do with a set of weights, and they make great foundations for new stuff, but yeah we may never see a competitive model that's 100% buildable at home. Edit…
"Open weights" means you can use the weights for free (as in beer). "Open source" means you get the training dataset and the methodology. ~Nobody does open source LLMs.
Using open data and dclm: https://github.com/mlfoundations/dclm
Earlier quoted context omitted.
Linux is open source and is mostly C code. You cannot run C code directly, you have to compile it and produce binaries. But it's the C code, not binary form, where the collaboration happens. With LLMs, weights are the binary code: it's how you run the model. But to be able to train the model from scratch, or to collaborate on new approaches, you have to operate at a the level of architecture, methods, and training da…
Analogies are always going to fall short. With LLM weights, you can modify them (quant, fine-tuning) to get something different, which is not something you do with compiled binaries. There are ample areas for collaboration even without being able to reproduce from scratch, which takes $X Millions of dollars, also something that a typical binary does not have as a feature.
Earlier quoted context omitted.
Honest question. As far as LLMs are concerned, isn't open weights same as open source?
No, I would argue that from the three main ingredients - training data, model source code and weights - weights are the furthest away from something akin to source code. They're more like obfuscated binaries. When it comes to fine-tuning only however things shift a little bit, yes.
“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…
AI is a fad, the brick and mortar of the future is open source tools.
Earlier quoted context omitted.
Doubtful that GPUs purchased today would be in use for a similar time scale. Govt investment would also drive the cost of GPUs up a great deal. Not sure why a publicly accessible GPU cluster would be a better solution than the current system of research grants.
> Doubtful that GPUs purchased today would be in use for a similar time scale Totally agree. That doesn't mean it can't generate massive ROI. > Govt investment would also drive the cost of GPUs up a great deal Difficult to say this ex ante . On its own, yes. But it would displace some demand. And it could help boost chip production in the long run. > Not sure why a publicly accessible GPU cluster would be a better so…
Along similar lines, I'm trying to build a developer credits program where I get whomever (AMD/Dell) to purchase credits on my super computers, that we then give away to developers to build solutions, which drives more demand for our hardware, and we commit to re-invest those credits back into more hardware. The idea is to create a win-win-win (us, them, you) developer flywheel ecosystem. It isn't a new idea at all, Nvidia and hyperscalers have been doing this for ages.
“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…
How about using some of that money to develop CUDA alternatives so everyone is not paying the Nvidia tax?