Open source AI is the path forward • Hacker News Board

Re: Open source AI is the path forward

#21

Post by redleader55 » Tue, Jul 23, 2024, 3:40 PM UTC

amusingimpala75 wrote: post #2

Sure but under what license? Because slapping “open source” on the model doesn’t make it open source if it’s not actually license that way. The 3.1 license still contains their non-commercial clause (over 700m users) and requires derivatives, whether fine tunings or trained on generated data, to use the llama name.

"Use it for whatever you want(conditions apply), but not if you are Google, Amazon, etc. If you become big enough talk to us." That's how I read the license, but obviously I might be missing some nuance.

Re: Open source AI is the path forward

#22

Post by CardenB » Tue, Jul 23, 2024, 3:40 PM UTC

JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

Doubtful that GPUs purchased today would be in use for a similar time scale. Govt investment would also drive the cost of GPUs up a great deal.

Not sure why a publicly accessible GPU cluster would be a better solution than the current system of research grants.

Re: Open source AI is the path forward

#23

Post by LorenDB » Tue, Jul 23, 2024, 3:40 PM UTC

ldjkfkdsjnv wrote: post #9

On every post like this on hacker news, I post my pro meta speech. My opinion has always been the same: 1. Meta pushed engineering wages higher across the industry. 2. They promote high performing engineers very quickly. There are engineers making 7 figures there with just a few years experience. 3. They have open sourced the most important frameworks: React and Pytorch This company is a guiding light forcing the han…

I never really liked Zuck due to Facebook's massive black hole of user tracking, but you've gotta give it to him: the Llama models are insanely good, and the commitment to making Horizon OS available to other VR manufacturers is commendable, although I'd like to see it truly open sourced.

Re: Open source AI is the path forward

#24

Post by orthoxerox » Tue, Jul 23, 2024, 3:40 PM UTC

the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

Open training dataset + open steps sufficient to train exactly the same model.

Re: Open source AI is the path forward

#25

Post by ChadNauseam » Tue, Jul 23, 2024, 3:40 PM UTC

mesebrec wrote: post #16

Note that Meta's models are not open source in any interpretation of the term. * You can't use them for any purpose. For example, the license prohibits using these models to train other models. * You can't meaningfully modify them given there is almost no information available about the training data, how they were trained, or how the training data was processed. As such, the model itself is not available under an op…

> you can't meaningfully modify them given there is almost no information available about the training data, how they were trained, or how the training data was processed.

I was under the impression that you could still fine-tune the models or apply your own RLHF on top of them. My understanding is that the training data would mostly be useful for training the model yourself from scratch (possibly after modifying the training data), which would be extremely expensive and out of reach for most people

Re: Open source AI is the path forward

#26

Post by bilsbie » Tue, Jul 23, 2024, 3:41 PM UTC

the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

Can’t you do fine tuning on those binaries? That’s a modification.

Re: Open source AI is the path forward

#27

Post by abetusk » Tue, Jul 23, 2024, 3:41 PM UTC

Another case of "open-washing". Llama is not available open source, under the common definition of open source, as the license doesn't allow for commercial re-use by default [0].

They provide their model, with weights and code, as "source available" and it looks like they allow for commercial use until a 700M monthly subscriber cap is surpassed. They also don't allow you to train other AI models with their model:

""" ... v. You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Meta Llama 3 or derivative works thereof). ... """

[0] https://github.com/meta-llama/llama3/blob/main/LICENSE

Re: Open source AI is the path forward

#28

Post by [deleted] » Tue, Jul 23, 2024, 3:41 PM UTC

ldjkfkdsjnv wrote: post #9

On every post like this on hacker news, I post my pro meta speech. My opinion has always been the same: 1. Meta pushed engineering wages higher across the industry. 2. They promote high performing engineers very quickly. There are engineers making 7 figures there with just a few years experience. 3. They have open sourced the most important frameworks: React and Pytorch This company is a guiding light forcing the han…

[deleted]

Re: Open source AI is the path forward

#29

Post by jsheard » Tue, Jul 23, 2024, 3:41 PM UTC

the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

I also think that something like Chromium is a better analogy for corporate open source models than a grassroots project like Linux is. Chromium is technically open source, but Google has absolute control over the direction of it's development and realistically it's far too complex to maintain a fork without Googles resources, just like Meta has complete control over what goes into their open models, and even if they did release all the training data and code (which they don't) us mere plebs could never afford to train a fork from scratch anyway.

Re: Open source AI is the path forward

#30

Post by JumpCrisscross » Tue, Jul 23, 2024, 3:42 PM UTC

CardenB wrote: post #22

JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

Doubtful that GPUs purchased today would be in use for a similar time scale. Govt investment would also drive the cost of GPUs up a great deal. Not sure why a publicly accessible GPU cluster would be a better solution than the current system of research grants.

> Doubtful that GPUs purchased today would be in use for a similar time scale

Totally agree. That doesn't mean it can't generate massive ROI.

> Govt investment would also drive the cost of GPUs up a great deal

Difficult to say this ex ante. On its own, yes. But it would displace some demand. And it could help boost chip production in the long run.

> Not sure why a publicly accessible GPU cluster would be a better solution than the current system of research grants

Those receiving the grants have to pay a private owner of the GPUs. That gatekeeping might be both problematic, if there is a conflict of interests, and inefficient. (Consider why the government runs its own supercomputers versus contracting everything to Oracle and IBM.)