Open source AI is the path forward • Hacker News Board

Re: Open source AI is the path forward

#471

Post by moffkalast » Tue, Jul 23, 2024, 8:55 PM UTC

Zambyte wrote: post #401

moffkalast wrote: post #387

Earlier quoted context omitted.

You can find the entire Llama 3.0 pretraining set here: https://huggingface.co/datasets/HuggingFaceFW/fineweb 15T tokens, 45 terrabytes. Seems fairly open source to me.

Where has Facebook linked that? I can't find anywhere that they actually published that.

Yeah I don't think I've seen it linked officially, but Meta does this sort of semi-official stuff all the time, leaking models ahead of time for PR, they even have a dedicated Reddit account for releasing unofficial info.

Regardless, it fits the compute used and the claim that they trained from public web data, and was suspiciously published by HF staff shortly after L3 released. It's about as official as the Mistral 7B v0.2 base model. I.e. mostly, but not entirely, probably for some weird legal reasons.

Re: Open source AI is the path forward

#472

Post by jononor » Tue, Jul 23, 2024, 8:55 PM UTC

JimDabell wrote: post #205

exe34 wrote: post #71

Earlier quoted context omitted.

the training data is the source.

I don’t think it’s that simple. The source is “the preferred form of the work for making modifications to it” (to use the GPL’s wording). For an LLM, that’s not the training data. That’s the model itself. You don’t make changes to an LLM by going back to the training data and making changes to it, then re-running the training. You update the model itself with more training data. You can’t even use the training code a…

Given a well behaved training setup, you will give an equivalently powerful model, given the same dataset and training scripts, and training settings. At least if you are willing to run it several times, an pick the best one - a process that is commonly used for large models.

Re: Open source AI is the path forward

#473

Post by blackeyeblitzar » Tue, Jul 23, 2024, 8:55 PM UTC

candiddevmike wrote: post #32

the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

None of Meta's models are "open source" in the FOSS sense, even the latest Llama 3.1. The license is restrictive. And no one has bothered to release their training data either. This post is an ad and trying to paint these things as something they aren't.

AI2 has released training data in their OLMo model: https://blog.allenai.org/hello-olmo-a-truly-open-llm-43f7e73...

Re: Open source AI is the path forward

#474

Post by nothrowaways » Tue, Jul 23, 2024, 8:55 PM UTC

the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

Weight is the new code.

Re: Open source AI is the path forward

#475

Post by j_maffe » Tue, Jul 23, 2024, 8:55 PM UTC

shnock wrote: post #420

baby wrote: post #380

Earlier quoted context omitted.

Meta does a good thing HN spends a day figuring out how it’s actually bad

It’s not actually bad, OP’s point is that it is not motivated by altruism. An action can be beneficial to the people without that effect being the incentive

Of course, it's not altruism; it's a publicly traded corporation. No one should ever believe in any such claims by these organizations. Non-altruistic organizations can still make positive-impact actions when they align with their goals.

Re: Open source AI is the path forward

#476

Post by [deleted] » Tue, Jul 23, 2024, 8:56 PM UTC

yjftsjthsd-h wrote: post #416

mkolodny wrote: post #252

Earlier quoted context omitted.

I disagree. The file I linked to, model.py, contains the Llama 3 model itself. You can use that model with open data to train it from scratch yourself. Or you can load Meta’s open weights and have a working LLM.

> The file I linked to, model.py, contains the Llama 3 model itself. That makes it source available ( https://en.wikipedia.org/wiki/Source-available_software ), not open source

[deleted]

Re: Open source AI is the path forward

#477

Post by bigmattystyles » Tue, Jul 23, 2024, 8:56 PM UTC

Am I too skeptical or is this the approach taken when they've decided they can't make proprietary work for them.

Re: Open source AI is the path forward

#478

Post by [deleted] » Tue, Jul 23, 2024, 8:57 PM UTC

Palomides wrote: post #364

phkahler wrote: post #302

Earlier quoted context omitted.

>> ...but this part strikes me as potentially anticompetitive reasoning. "Commoditize your complements" is an accepted strategy. And while pricing below cost to harm competitors is often illegal, the reality is that the marginal cost of software is zero.

spending a very quantifiable large amount of money to release something your nominal competitors charge for without having your own direct business case for it seems a little much

[deleted]

Re: Open source AI is the path forward

#479

Post by nothrowaways » Tue, Jul 23, 2024, 8:59 PM UTC

I love llamas but how fb can profit from leading this effort is not clear to me.

Re: Open source AI is the path forward

#480

Post by kemiller » Tue, Jul 23, 2024, 8:59 PM UTC

halflings wrote: post #465

the8thbit wrote: post #12

"Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ..." I find the language around "open source AI" to be confusing. With "open source" there's usually "source" to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They're very large matric…

Training code is only useful to people in academia, and the closest thing to "code you can modify" are open weights. People are framing this as if it was an open-source hierarchy, with "actual" open-source requiring all training code to be shared. This is not obvious to me, as I'm not asking people that share open-source libraries to also share the tools they used to develop them. I'm also not asking them to share al…

I dunno — if an open source project required, say, a proprietary compiler, that would diminish its open source-ness. But I agree it's not totally comparable, since the weights are not particularly analogous to machine code. We probably need a new term. Open Weights.