Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source AI is the path forward

about.fb.com

Post reply on HN
Print view View original

101–110 of 936 posts

Re: Open source AI is the path forward

#101

Post by orthoxerox »

the8thbit wrote: post #51
orthoxerox wrote: post #24

Earlier quoted context omitted.

Open training dataset + open steps sufficient to train exactly the same model.

This isn't what Meta releases with their models, though I would like to see more public training data. However, I still don't think that would qualify as "open source". Something isn't open source just because its reproducible out of composable parts. If one, very critical and system defining part is a binary (or similar) without publicly available source code, then I don't think it can be said to be "open source". T…

That's what I meant by "open steps", I guess I wasn't clear enough.

Re: Open source AI is the path forward

#102

Post by aloe_falsa »

sidcool wrote: post #69
mvkel wrote: post #46

It's a real shame that we're still calling Llama "open source" when at best it's "open weights." Not that anyone would go buy 100,000 H100s to train their own Llama, but words matter. Definitions matter.

Honest question. As far as LLMs are concerned, isn't open weights same as open source?

GPL defines the “source code” of a work as the preferred form of the work for making modifications to it. If Meta released a petabyte of raw training data, would that really be easier to extend and adapt (as opposed to fine-tuning the weights)?

Re: Open source AI is the path forward

#103

Post by [deleted] »

stale2002 wrote: post #82
the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

Ok call it Open Weights then if the dictionary definitions matter so much to you. The actual point that matters is that these models are available for most people to use for a lot of stuff, and this is way way better than what competitors like OpenAI offer.

[deleted]

Re: Open source AI is the path forward

#104

Post by diggan »

> Today we’re taking the next steps towards open source AI becoming the industry standard. We’re releasing Llama 3.1 405B, the first frontier-level open source AI model,

Why do people keep mislabeling this as Open Source? The whole point of calling something Open Source is that the "magic sauce" of how to build something is publicly available, so I could built it myself if I have the means. But without the training data publicly available, could I train Llama 3.1 if I had the means? No wonder Zuckerberg doesn't start with defining what Open Source actually means, as then the blogpost would have lost all meaning from the get go.

Just call it "Open Model" or something. As it stands right now, the meaning of Open Source is being diluted by all these companies pretending to doing one thing, while actually doing something else.

I initially got very exciting seeing the title and the domain, but hopelessly sad after reading through the article and realizing they're still trying to pass their artifacts off as Open Source projects.

Re: Open source AI is the path forward

#105

Post by stale2002 »

tw04 wrote: post #63
JumpCrisscross wrote: post #20

Earlier quoted context omitted.

> if Facebook were selling their AI advances instead of selling the output it produces, they wouldn't be advocating for everyone else to open source their stack You're acting as if commoditizing one's complements is either new or reprehensible [1]. [1] https://gwern.net/complement

>You're acting as if commoditizing one's complements is either new or reprehensible [1]. I'm acting as if calling on other companies to open source their core product, just because it's a complement for you, and acting as if it's for the benefit of mankind is disingenuous, which it is.

> as if it's for the benefit of mankind

But it does benefit mankind.

More free tech products is good for the world.

This is a good thing. When people or companies do good things, they should get the credit for doing good things.

Re: Open source AI is the path forward

#106

Post by patrickaljord »

blackeyeblitzar wrote: post #81
mkolodny wrote: post #73

Earlier quoted context omitted.

Llama’s code is open source: https://github.com/meta-llama/llama3/blob/main/llama/model.p...

That is just the inference code. Not training code or evaluation code or whatever pre/post processing they do.

Is there an LLM with actual open source training code and dataset? Besides BLOOM https://huggingface.co/bigscience/bloom

Re: Open source AI is the path forward

#107

Post by diggan »

JumpCrisscross wrote: post #84
exe34 wrote: post #71

Earlier quoted context omitted.

the training data is the source.

> the training data is the source Sure. But that's not going to be released. The term open source AI cannot be expected to cover it because it's not practical.

So because it's really hard to do proper Open Source with these LLMs, means we need to change the meaning of Open Source so it fits with these PR releases?

Re: Open source AI is the path forward

#108

Post by paulhilbert »

sidcool wrote: post #69
mvkel wrote: post #46

It's a real shame that we're still calling Llama "open source" when at best it's "open weights." Not that anyone would go buy 100,000 H100s to train their own Llama, but words matter. Definitions matter.

Honest question. As far as LLMs are concerned, isn't open weights same as open source?

No, I would argue that from the three main ingredients - training data, model source code and weights - weights are the furthest away from something akin to source code.

They're more like obfuscated binaries. When it comes to fine-tuning only however things shift a little bit, yes.

Re: Open source AI is the path forward

#109

Post by JumpCrisscross »

tw04 wrote: post #63
JumpCrisscross wrote: post #20

Earlier quoted context omitted.

> if Facebook were selling their AI advances instead of selling the output it produces, they wouldn't be advocating for everyone else to open source their stack You're acting as if commoditizing one's complements is either new or reprehensible [1]. [1] https://gwern.net/complement

>You're acting as if commoditizing one's complements is either new or reprehensible [1]. I'm acting as if calling on other companies to open source their core product, just because it's a complement for you, and acting as if it's for the benefit of mankind is disingenuous, which it is.

> acting as if it's for the benefit of mankind is disingenuous, which it is

Is it bad for mankind that Meta publishes its weights? Mutually beneficial is a valid game state--there is no moral law that requires anything good be made as a sacrifice.

Re: Open source AI is the path forward

#110

Post by the8thbit »

apsec112 wrote: post #80
mkolodny wrote: post #73

Earlier quoted context omitted.

Llama’s code is open source: https://github.com/meta-llama/llama3/blob/main/llama/model.p...

That's not the training code, just the inference code. The training code, running on thousands of high-end H100 servers, is surely much more complex. They also don't open-source the dataset, or the code they used for data scraping/filtering/etc.

"just the inference code"

It's not the "inference code", its the code that specifies the architecture of the model and loads the model. The "inference code" is mostly the model, and the model is not legible to a human reader.

Maybe someday open source models will be possible, but we will need much better interpretability tools so we can generate the source code from the model. In most software projects you write the source as a specification that is then used by the computer to implement the software, but in this case the process is reversed.

Post reply on HN