Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source AI is the path forward

about.fb.com

Post reply on HN
Print view View original

191–200 of 936 posts

Re: Open source AI is the path forward

#191

Post by [deleted] »

candiddevmike wrote: post #32
the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

None of Meta's models are "open source" in the FOSS sense, even the latest Llama 3.1. The license is restrictive. And no one has bothered to release their training data either. This post is an ad and trying to paint these things as something they aren't.

[deleted]

Re: Open source AI is the path forward

#192

Post by whimsicalism »

rvnx wrote: post #148
Teever wrote: post #88

Earlier quoted context omitted.

There was a post[0] on here recently about how the US went from producing woefully insufficient numbers of aircraft to producing 300k by the end of world war 2. One of the things that the post mentioned was the meager profit margin that the companies made during this time. But the thing is that this set the America auto and aviation industry up to rule the world for decades. A government going to a company and saying…

In the case of NVIDIA it's even more sneaky. They are an intellectual property company holding the rights on plans to make graphic cards, not even a company actually making graphic cards. The government could launch an initiative "OpenGPU" or "OpenAI Accelerator", where the government orders GPUs from TSMC directly, without the middleman. It may require some tweaking in the law to allow exception to intellectual prop…

y'all really don't understand how these actions would seriously harm capital markets and make it difficult for private capital formation to produce innovations going forward.

Re: Open source AI is the path forward

#193

Post by enriquto »

talldayo wrote: post #174
diggan wrote: post #151

Earlier quoted context omitted.

Not just the training code but the training data as well, should be under a permissive license, otherwise you cannot call the project itself Open Source, which Facebook does here. > is way less valuable than the weights for the vast majority of people The same is true for most Open Source projects, most people use the distributed binaries or other artifacts from the projects, and couldn't care less about the code its…

I will steelman the idea that a tokenizer and weights are all you need for the "source" of an LLM. They are components that can be modified, redistributed and when put together, reproduce the full experience intended. If we insist upon the release of training data with Open models, you might as well kiss the idea of usable Open LLMs out the door. Most of the content in training datasets like The Pile are not licensed…

> Most of the content in training datasets like The Pile are not licensed for redistribution in any way shape or form.

But distributing the weights is a "form" of distribution. You can recover many items of the dataset (most easily, the outliers) by using the weights.

Just because they are codified in a non-readily accessible way, does not mean that you are not distributing them.

It's scary to think that "training" is becoming a thinly veiled way to strip copyright of works.

Re: Open source AI is the path forward

#194

Post by jncfhnb »

diggan wrote: post #151
valine wrote: post #135

Earlier quoted context omitted.

The codebase to do the training is way less valuable than the weights for the vast majority of people. Releasing the training code would be nice, but it doesn't really help anyone but Meta's direct competitors. If you want to train on top of Llama there's absolutely nothing stopping you. Plenty of open source tools to do parameter optimization.

Not just the training code but the training data as well, should be under a permissive license, otherwise you cannot call the project itself Open Source, which Facebook does here. > is way less valuable than the weights for the vast majority of people The same is true for most Open Source projects, most people use the distributed binaries or other artifacts from the projects, and couldn't care less about the code its…

People don’t typically modify distributed binaries.

People do typically modify model weights. They are the preferred form to modify model.

Saying “build” llama is just a nonsense comparison to traditional compiled software. “Building llama” is more akin to taking the raw weights as text and putting them into a nice pickle file. Or loading it into an inference engine.

Demanding that you have everything needed to recreate the weights from scratch is like arguing an application cannot be open source unless it also includes the user testing history and design documents.

And of course some idiots don’t understand what a pickled weights file is and claim it’s as useless as a distributed binary if you want to modify the program just because it is technically compiled; not understanding that the point of the pickled file is “convenience” and that it unpacks back to the original form. Like arguing open source software can’t be distributed in zip files.

> Say I have a farm of 100 A100's, could I reproduce the Llama model from scratch today?

Say you have a piece of paper. Can you reproduce `print(“hello world”)` from scratch?

Re: Open source AI is the path forward

#195

Post by [deleted] »

amelius wrote: post #128

> One of my [Mark Zuckerberg, ed.] formative experiences has been building our services constrained by what Apple will let us build on their platforms. Between the way they tax developers, the arbitrary rules they apply, and all the product innovations they block from shipping, it’s clear that Meta and many other companies would be freed up to build much better services for people if we could build the best versions…

[deleted]

Re: Open source AI is the path forward

#196

Post by bornfreddy »

unshavedyak wrote: post #165
diggan wrote: post #151

Earlier quoted context omitted.

Not just the training code but the training data as well, should be under a permissive license, otherwise you cannot call the project itself Open Source, which Facebook does here. > is way less valuable than the weights for the vast majority of people The same is true for most Open Source projects, most people use the distributed binaries or other artifacts from the projects, and couldn't care less about the code its…

> Not just the training code but the training data as well, should be under a permissive license, otherwise you cannot call the project itself Open Source, which Facebook does here. Does FB even have the capability to do that? I'd assume there's a bunch of data that's not theirs and they can't even release it. Let alone some data that they might not want to admit is in the source.

If not, it is questionable if they should train on such data anyway.

Also, that doesn't matter in this discussion - if you are unable to release the source under appropriate licence (for whatever reason), you should not call it Open Source.

Re: Open source AI is the path forward

#197

Post by jdminhbg »

thenoblesunfish wrote: post #182
jdminhbg wrote: post #169

Earlier quoted context omitted.

> Why do people keep mislabeling this as Open Source? The whole point of calling something Open Source is that the "magic sauce" of how to build something is publicly available, so I could built it myself if I have the means. But without the training data publicly available, could I train Llama 3.1 if I had the means? I don't think not releasing the commit history of a project makes it not Open Source, this seems lik…

You don't need to have the commit history to see "how it works". ML that works well does so in huge part due to the training data used. The leading models today aren't distinguished by the way they're trained, but what they're trained on.

I agree that you need training data to build AI from scratch, much like you need lots of really smart developers and a mailing list and servers and stuff to build the Linux kernel from scratch. But it's not like having the training data and training code will get you the same result, in the way something like open data in science is about replicating results.

Re: Open source AI is the path forward

#199

Post by goda90 »

JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

I'd like to see big programs to increase the amount of cheap, clean energy we have. AI compute would be one of many beneficiaries of super cheap energy, especially since you wouldn't need to chase newer, more efficient hardware just to keep costs down.

Re: Open source AI is the path forward

#200

Post by diggan »

JumpCrisscross wrote: post #171
Palomides wrote: post #125

Earlier quoted context omitted.

source available is absolutely not the same as open source you are playing very loosely with terms that have specific, widely accepted definitions (e.g. https://opensource.org/osd ) I don't get why you think it would be useful to call LLMs with published weights "open source"

> terms that have specific, widely accepted definitions OSF's definition is far from the only one [1]. Switzerland is currently implementing CH Open's definition, the EU another one, et cetera . > I don't get why you think it would be useful to call LLMs with published weights "open source" I don't. I'm saying that if the choice is between open weights or open weights + open training data, open weights will win becau…

For the EU, I'm guessing you're talking about the EUPL, which is FSF/OSI approved and GPL compatible, generally considered copyleft.

For the CH Open, I'm not finding anything specific, even from Swiss websites, could you help me understand what you're referring to here?

I'm guessing that all these definitions have at least some points in common, which involves (another guess) at least being able to produce the output artifacts/binaries by yourself, something that you cannot do with Llama, just as an example.

Post reply on HN