Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source AI is the path forward

about.fb.com

Post reply on HN
Print view View original

51–60 of 936 posts

Re: Open source AI is the path forward

#51

Post by the8thbit »

orthoxerox wrote: post #24
the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

Open training dataset + open steps sufficient to train exactly the same model.

This isn't what Meta releases with their models, though I would like to see more public training data. However, I still don't think that would qualify as "open source". Something isn't open source just because its reproducible out of composable parts. If one, very critical and system defining part is a binary (or similar) without publicly available source code, then I don't think it can be said to be "open source". That would be like saying that Windows 11 is open source because Windows Calculator is open source, and its a component of Windows.

Re: Open source AI is the path forward

#52

Post by fweimer »

JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

Don't these public clusters exist today, and have been around for decades at this point, with varying architectures? In the sense that you submit a proposal, it gets approved, and then you get access for your research?

Re: Open source AI is the path forward

#53

Post by blackeyeblitzar »

JumpCrisscross wrote: post #15
blackeyeblitzar wrote: post #6

Only if it is truly open source (open data sets, transparent curation/moderation/censorship of data sets, open training source code, open evaluation suites, and an OSI approved open source license). Open weights (and open inference code) is NOT open source, but just some weak open washing marketing. The model that comes closest to being TRULY open is AI2’s OLMo. See their blog post on their approach: https://blog.all…

> Only if it is truly open source (open data sets, transparent curation/moderation/censorship of data sets, open training source code, open evaluation suites, and an OSI approved open source license) You’re missing a then to your if. What happens if it’s “truly” open per your definition versus not?

I think you are asking what the benefits are? The main benefit is that we can trust what these systems are doing better. Or we can self host them. If we just take the weights, then it is unclear how these systems might be lying to us or manipulating us.

Another benefit is that we can learn from how the training and other steps actually work. We can change them to suit our needs (although costs are impractical today). Etc. It’s all the usual open source benefits.

Re: Open source AI is the path forward

#54

Post by rybosworld »

Huge companies like facebook will often argue for solutions that on the surface, seem to be in the public interest.

But I have strong doubts they (or any other company) actually believe what they are saying.

Here is the reality:

- Facebook is spending untold billions on GPU hardware.

- Facebook is arguing in favor of open sourcing the models, that they spent billions of dollars to generate, for free...?

It follows that companies with much smaller resources (money) will not be able to match what Facebook is doing. Seems like an attempt to kill off the competition (specifically, smaller organizations) before they can take root.

Re: Open source AI is the path forward

#55

Post by mesebrec »

ChadNauseam wrote: post #25
mesebrec wrote: post #16

Note that Meta's models are not open source in any interpretation of the term. * You can't use them for any purpose. For example, the license prohibits using these models to train other models. * You can't meaningfully modify them given there is almost no information available about the training data, how they were trained, or how the training data was processed. As such, the model itself is not available under an op…

> you can't meaningfully modify them given there is almost no information available about the training data, how they were trained, or how the training data was processed. I was under the impression that you could still fine-tune the models or apply your own RLHF on top of them. My understanding is that the training data would mostly be useful for training the model yourself from scratch (possibly after modifying the…

Indeed, fine-tuning is still possible, but you can only go so far with fine-tuning before you need to completely retrain the model.

This is why Silo AI, for example, had to start from scratch to get better support for small European languages.

Re: Open source AI is the path forward

#56

Post by epaulson »

JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

The National Science Foundation has been doing this for decades, starting with the supercomputing centers in the 80s. Long before anyone talked about cloud credits, NSF has had a bunch of different programs to allocate time on supercomputers to researchers at no cost, these days mostly run out of the Office of Advanced Cyberinfrastruture. (The office name is from the early 00s) - https://new.nsf.gov/cise/oac

(To connect universities to the different supercomputing centers, the NSF funded the NSFnet network in the 80s, which was basically the backbone of the Internet in the 80s and early 90s. The supercomputing funding has really, really paid off for the USA)

Re: Open source AI is the path forward

#57

Post by [deleted] »

JumpCrisscross wrote: post #15
blackeyeblitzar wrote: post #6

Only if it is truly open source (open data sets, transparent curation/moderation/censorship of data sets, open training source code, open evaluation suites, and an OSI approved open source license). Open weights (and open inference code) is NOT open source, but just some weak open washing marketing. The model that comes closest to being TRULY open is AI2’s OLMo. See their blog post on their approach: https://blog.all…

> Only if it is truly open source (open data sets, transparent curation/moderation/censorship of data sets, open training source code, open evaluation suites, and an OSI approved open source license) You’re missing a then to your if. What happens if it’s “truly” open per your definition versus not?

[deleted]

Re: Open source AI is the path forward

#58

Post by JumpCrisscross »

light_hue_1 wrote: post #48
JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

The problem is that any public cluster would be outdated in 2 years. At the same time, GPUs are massively overpriced. Nvidia's profit margins on the H100 are crazy. Until we get cheaper cards that stand the test of time, building a public cluster is just a waste of money. There are far better ways to spend $1b in research dollars.

> any public cluster would be outdated in 2 years

The private companies buying hundreds of billions of dollars of GPUs aren't writing them off in 2 years. They won't be cutting edge for long. But that's not the point--they'll still be available.

> Nvidia's profit margins on the H100 are crazy

I don't see how the current practice of giving a researcher a grant so they can rent time on a Google cluster that runs H100s is more efficient. It's just a question of capex or opex. As a state, the U.S. has a structual advantage in the former.

> far better ways to spend $1b in research dollars

One assumes the U.S. government wouldn't be paying list price. In any case, the purpose isn't purely research ROI. Like the heavy presses, it's in making a prohibitively-expensive capital asset generally available.

Re: Open source AI is the path forward

#59

Post by JumpCrisscross »

epaulson wrote: post #56
JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

The National Science Foundation has been doing this for decades, starting with the supercomputing centers in the 80s. Long before anyone talked about cloud credits, NSF has had a bunch of different programs to allocate time on supercomputers to researchers at no cost, these days mostly run out of the Office of Advanced Cyberinfrastruture. (The office name is from the early 00s) - https://new.nsf.gov/cise/oac (To conn…

> NSF has had a bunch of different programs to allocate time on supercomputers to researchers at no cost, these days mostly run out of the Office of Advanced Cyberinfrastruture

This would be the logical place to put such a programme.

Re: Open source AI is the path forward

#60

Post by JumpCrisscross »

fweimer wrote: post #52
JumpCrisscross wrote: post #7

“The Heavy Press Program was a Cold War-era program of the United States Air Force to build the largest forging presses and extrusion presses in the world.” This ”program began in 1944 and concluded in 1957 after construction of four forging presses and six extruders, at an overall cost of $279 million. Six of them are still in operation today, manufacturing structural parts for military and commercial aircraft” [1].…

Don't these public clusters exist today, and have been around for decades at this point, with varying architectures? In the sense that you submit a proposal, it gets approved, and then you get access for your research?

Not--to my knowledge--for the GPUs necessary to train cutting-edge LLMs.
Post reply on HN