Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source AI is the path forward

about.fb.com

Post reply on HN
Print view View original

461–470 of 936 posts

Re: Open source AI is the path forward

#461

Post by wanderingbort »

celdon25 wrote: post #404
advael wrote: post #367

Earlier quoted context omitted.

As an ML researcher, I agree. Meta doesn't include adequate information to replicate the models, and from the perspective of fundamental research, the interest that big tech companies have taken in this field has been a significant impediment to independent researchers, despite the fact that they are undeniably producing groundbreaking results in many respects, due to this fundamental lack of openness This should als…

Which option would be better? A) Release the data, and if it ends up causing a privacy scandal, at least you can actually call it open this time. B) Neuter the dataset, and the model All I ever see in these threads is a lot of whining and no viable alternative solutions (I’m fine with the idea of it being a hard problem, but when I see this attitude from “researchers” it makes me less optimistic about the future) > a…

> Release the data, and if it ends up causing a privacy scandal...

We can't prove that a model like llama will never produce a segment of its training data set verbatim.

Any potential privacy scandal is already in motion.

My cynical assumption is that Meta knows that competitors like OpenAI have PR-bombs in their trained model and therefore would never opensource the weights.

Re: Open source AI is the path forward

#462

Post by gorgoiler »

the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

One counterpoint is that major publications (eg New York Times) would have you believe that AI is a mildly lossy compression algorithm capable of reconstructing the original source material.

Re: Open source AI is the path forward

#463

Post by Palmik »

rednafi wrote: post #454

How’s only sharing the binary artifact is open source? There’s the data aspect of things that they can’t share because of licensing and the code itself isn’t accessible.

It's much better than sharing a binary artifact of regular software, since the weights can be and are easily and frequently modified by fine tuning the model. This means you can modify the "binary artifact" to your needs, similar to how you might change the code of open source software to add features etc.

Re: Open source AI is the path forward

#464

Post by jononor »

sailingparrot wrote: post #402
wrs wrote: post #323

Earlier quoted context omitted.

Calling that “open source” renders the word “source” meaningless. By your definition, I can release a binary executable freely and call it “open source” because you can modify it to do whatever you want. Model weights are like a binary that nobody has the source for. We need another term.

No it’s not the same as releasing a binary, feels like we can’t get out of the pedantics. I can in theory modify a binary to do whatever I want. In practice it is intractably hard to make any significant modification to a binary, and even if you could, you would then not be legally allowed to e.g. redistribute. Here, modifying that model is not harder that doing regular ML, and I can redistribute. Meta doesn’t have a…

You have to write all the code needed to do the modifications you are interested in. That is, there is no source code provided that can be used to make the modifications of interest. One also has to come up with suite le datasets, from scratch. Training setup and data is completely non trivial for a large language model. To replicate Llama would take hundreds of hours of engineering, at least.

Re: Open source AI is the path forward

#465

Post by halflings »

the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

Training code is only useful to people in academia, and the closest thing to "code you can modify" are open weights.

People are framing this as if it was an open-source hierarchy, with "actual" open-source requiring all training code to be shared. This is not obvious to me, as I'm not asking people that share open-source libraries to also share the tools they used to develop them. I'm also not asking them to share all the design documents/architecture discussion behind this software. It's sufficient that I can take the end result and reshape it in any way I desire.

This is coming from an LLM practitioner that finetunes models for a living; and this constant debate about open-source vs open-weights seems like a huge distraction vs the impact open-sourcing something like Llama has... this is truly a Linux-like moment. (at a much smaller scale of course, for now at least)

Re: Open source AI is the path forward

#466

Post by sebastiennight »

gowld wrote: post #460
aerzen wrote: post #422

Earlier quoted context omitted.

LLAMA is an open-weights model. I like this term, let's use that instead of open source.

Can a human programmer edit the weights according to some semantics?

It is possible to merge two fine-tunes of models from the same family by... wait for it... averaging or combining their weights[0].

I am still amazed that we can do that.

[0]: https://arxiv.org/abs/2212.09849

Re: Open source AI is the path forward

#467

Post by Palmik »

tmsh wrote: post #431

Software 2.0 is about open licensing. I.e., the more important thing - the more "free" thing - is the licensing now. E.g., I play around with different image diffusion models like Stable Diffusion and specific fine-tuned variations for ControlNet or LoRA that I plug into ComfyUI. But I can't use it at work because of the licensing. I have to use InvokeAI instead of ComfyUI if I want to be careful and only very specif…

In the LLM world there are many open source solutions to find tuning, maybe the best one being from Meta: https://github.com/pytorch/torchtune

In terms of inference and interface (since you mentioned comfy) there are many truly open source options such as vLLM (though there isn't a single really performant open source solution for inference yet).

Re: Open source AI is the path forward

#468

Post by plausibility »

stavros wrote: post #449
sigmoid10 wrote: post #448

Earlier quoted context omitted.

>Nobody does open source LLMs. There are a bunch of independent, fully open source foundation models from companies that share everything (including all data). AMBER and MAP-NEO for example. But we have yet to see one in the 100B+ parameter category.

Sorry, the tilde before "nobody" is my notation for "basically nobody" or "almost nobody". I thought it was more common.

It is more common when it comes to numbers I guess. There are ~5 ancestors in this comment chain, if I would agree roughly 4-6 is acceptable.

Re: Open source AI is the path forward

#469

Post by gowld »

downWidOutaFite wrote: post #384
input_sh wrote: post #170

Earlier quoted context omitted.

Open Source Initiative (kind of a de-facto authority on what's open source and what not) is spending a whole lot of time figuring out what it means for an AI system to be open source. In other words, they're basically trying to come up with a new license because the existing ones can't easily apply. I believe this is the current draft: https://opensource.org/deepdive/drafts/the-open-source-ai-de...

OSI made themselves the authority because they hated Richard Stallman and his Free Software movement. It's just marketing.

RMS has no interest in governing Open Source, so your comment bears no particular relevance.

RMS is an advocate for Free Software. Free Software generally implies Open Source, but not the converse.

RMS considers openness of source to be a separate category from the freeness of software. "Free software is a political movement; open source is a development model."

https://www.gnu.org/licenses/license-list.en.html

Re: Open source AI is the path forward

#470

Post by blackeyeblitzar »

sidcool wrote: post #69
mvkel wrote: post #46

It's a real shame that we're still calling Llama "open source" when at best it's "open weights." Not that anyone would go buy 100,000 H100s to train their own Llama, but words matter. Definitions matter.

Honest question. As far as LLMs are concerned, isn't open weights same as open source?

No open weights are the output of a proprietary and secretive process of training. It’s like sharing a pre compiled application instead of what you need to reproduce the compiled application.

AI2’s OLMo is an example of what open source actually looks like for LLMs:

https://blog.allenai.org/hello-olmo-a-truly-open-llm-43f7e73...

Post reply on HN