Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source AI is the path forward

about.fb.com

Post reply on HN
Print view View original

481–490 of 936 posts

Re: Open source AI is the path forward

#481

Post by sailingparrot »

jononor wrote: post #464
sailingparrot wrote: post #402

Earlier quoted context omitted.

No it’s not the same as releasing a binary, feels like we can’t get out of the pedantics. I can in theory modify a binary to do whatever I want. In practice it is intractably hard to make any significant modification to a binary, and even if you could, you would then not be legally allowed to e.g. redistribute. Here, modifying that model is not harder that doing regular ML, and I can redistribute. Meta doesn’t have a…

You have to write all the code needed to do the modifications you are interested in. That is, there is no source code provided that can be used to make the modifications of interest. One also has to come up with suite le datasets, from scratch. Training setup and data is completely non trivial for a large language model. To replicate Llama would take hundreds of hours of engineering, at least.

> You have to write all the code needed to do the modifications you are interested in. That is, there is no source code provided that can be used to make the modifications of interest.

Just like open source?

> Training setup and data is completely non trivial for a large language model. To replicate Llama would take hundreds of hours of engineering, at least.

The entire point of having the pre-trained weight released is to *not* have to do this. You just need to finetune, which can be done with very little data, depending on the task, and many open source toolkits, that work with those weights, exist to make this trivial.

Re: Open source AI is the path forward

#482

Post by achrono »

_heimdall wrote: post #436
stavros wrote: post #425

Earlier quoted context omitted.

"Open weights" means you can use the weights for free (as in beer). "Open source" means you get the training dataset and the methodology. ~Nobody does open source LLMs.

Why is the dataset required for it to be open source? If I self host a project that is open sourced rather than paying for a hosted version, like Sentry.io for example, I don't expect data to come along with the code. Licensing rights are always up for debate in open source, but I wouldn't expect more than the code to be available and reviewable for anything needed to build and run the project. In the case of an LLM…

> not actually sure if Meta does share all that

Meta shares the code for inference but not for training, so even if we say it can be open-source without the training data, Meta's models are not open-source.

I can appreciate Zuck's enthusiasm for open-source but not his willingness to mislead the larger public about how open they actually are.

Re: Open source AI is the path forward

#483

Post by elevatedastalt »

diggan wrote: post #107
JumpCrisscross wrote: post #84

Earlier quoted context omitted.

> the training data is the source Sure. But that's not going to be released. The term open source AI cannot be expected to cover it because it's not practical.

So because it's really hard to do proper Open Source with these LLMs, means we need to change the meaning of Open Source so it fits with these PR releases?

No, we need to adapt an existing term into the new context that it is being deployed in.

Re: Open source AI is the path forward

#484

Post by jononor »

JimDabell wrote: post #205
exe34 wrote: post #71

Earlier quoted context omitted.

the training data is the source.

I don’t think it’s that simple. The source is “the preferred form of the work for making modifications to it” (to use the GPL’s wording). For an LLM, that’s not the training data. That’s the model itself. You don’t make changes to an LLM by going back to the training data and making changes to it, then re-running the training. You update the model itself with more training data. You can’t even use the training code a…

No the preferred way to make modifications is using the the training code. One may also input a snapshot weighs to start from, but the training code is definitely what you would modify to make a change.

Re: Open source AI is the path forward

#485

Post by choppaface »

hubraumhugo wrote: post #112

The big winners of this: devs and AI startups - No more vendor lock-in - Instead of just wrapping proprietary API endpoints, developers can now integrate AI deeply into their products in a very cost-effective and performant way - Price race to the bottom with near-instant LLM responses at very low prices are on the horizon As a founder, it feels like a very exciting time to build a startup as your product automatical…

Also the opportunity to run on user compute and on private data. That supports a slate of business models that are incompatible with the mainframe approach.

Including adtech models, which are predominantly cloud-based.

Re: Open source AI is the path forward

#486

Post by ab5tract »

causal wrote: post #41
the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

"Open weights" is a more appropriate term but I'll point out that these weights are also largely inscrutable to the people with the code that trained it. And for licensing reasons, the datasets may not be possible to share. There is still a lot of modifying you can do with a set of weights, and they make great foundations for new stuff, but yeah we may never see a competitive model that's 100% buildable at home. Edit…

If you can’t share the dataset, under what twisted reality are you fine to share the derivative models based on those unsharable datasets?

In a better world, there would be no “I ran some algos on it and now it’s mine” defense.

Re: Open source AI is the path forward

#487

Post by gowld »

the8thbit wrote: post #265
bilsbie wrote: post #153

Earlier quoted context omitted.

You make a good point but those are also just limitations of the technology (or at least our current understanding of it) Maybe an analogy would help. A family spent generations breeding the perfect apple tree and they decided to “open source” it. What would open sourcing look like?

"You make a good point but those are also just limitations of the technology (or at least our current understanding of it)" Yeah, that is my point. Things that don't have source code can't be open source. "Maybe an analogy would help. A family spent generations breeding the perfect apple tree and they decided to “open source” it. What would open sourcing look like?" I think we need to be weary of dilemmas without sol…

Be charitable and intellectually curious. What would "open" look like?

GNU says "The GNU GPL can be used for general data which is not software, as long as one can determine what the definition of “source code” refers to in the particular case. As it turns out, the DSL (see below) also requires that you determine what the “source code” is, using approximately the same definition that the GPL uses."

and offers these categories, for example:

https://www.gnu.org/licenses/license-list.en.html#NonFreeSof...

* Software Licenses

* * GPL-Compatible Free Software Licenses

\

* * GPL-Incompatible Free Software Licenses

\

* Licenses For Documentation

* * Free Documentation Licenses

\

* Licenses for Other Works

* * Licenses for Works of Practical Use besides Software and Documentation

* * Licenses for Fonts

* * Licenses for Works stating a Viewpoint (e.g., Opinion or Testimony)

* * Licenses for Designs for Physical Objects

Re: Open source AI is the path forward

#488

Post by pphysch »

thayne wrote: post #374
Zambyte wrote: post #238

Earlier quoted context omitted.

> If so, then how can current ML models be open source? The source of a language model is the text it was trained on. Llama models are not open source (contrary to their claims), they are open weight.

I think it would also include the code used to train it

That would be more analogous to the build toolchain than the source code, but yes

Re: Open source AI is the path forward

#489

Post by the8thbit »

sailingparrot wrote: post #402
wrs wrote: post #323

Earlier quoted context omitted.

Calling that “open source” renders the word “source” meaningless. By your definition, I can release a binary executable freely and call it “open source” because you can modify it to do whatever you want. Model weights are like a binary that nobody has the source for. We need another term.

No it’s not the same as releasing a binary, feels like we can’t get out of the pedantics. I can in theory modify a binary to do whatever I want. In practice it is intractably hard to make any significant modification to a binary, and even if you could, you would then not be legally allowed to e.g. redistribute. Here, modifying that model is not harder that doing regular ML, and I can redistribute. Meta doesn’t have a…

"In practice it is intractably hard to make any significant modification to a binary, and even if you could, you would then not be legally allowed to e.g. redistribute."

It depends on the binary and the license the binary is released under. If the binary is released to the public domain, for example, you are free to make whatever modifications you wish. And there are plenty of licenses like this, that allow closed source software to be used as the user wishes. That doesn't make it open source.

Likewise, there are plenty of closed source projects who's binaries we can poke and prod with much higher understanding of what our changes are actually doing than we're able to get when we poke and prod LLMs. If you want to make a Pokemon Red/Blue or Minecraft mod you have a lot of tools at your disposal.

A project that only exists as a binary which the copyright holder has relinquished rights to, or has released under some similar permissive closed source license, but people have poked around enough to figure out how to modify certain parts of the binary with some degree of predictability is a more apt analogy. Especially if the original author has lost the source code, as there is no source code the speak of when discussing these models.

I would not call that binary "open source", because the source would, in fact, not be open.

Re: Open source AI is the path forward

#490

Post by _heimdall »

stavros wrote: post #451
_heimdall wrote: post #436

Earlier quoted context omitted.

Why is the dataset required for it to be open source? If I self host a project that is open sourced rather than paying for a hosted version, like Sentry.io for example, I don't expect data to come along with the code. Licensing rights are always up for debate in open source, but I wouldn't expect more than the code to be available and reviewable for anything needed to build and run the project. In the case of an LLM…

Data is to models what code is to software.

I don't quite agree there. Based on other comments it sounds like Meta doesn't open source the code used to train the model, that would make it not open source in my book.

The trained model doesn't need to be open source though, and frankly I'm not sure what the value there is specifically with regards to OSS. I'm not aware of a solution to interpretability problem, even if the model is shared we can't understand what's in it.

Microsoft ships obfuscated code with Windows builds, but that doesn't make it open source.

Post reply on HN