401–410 of 936 posts
Earlier quoted context omitted.
> If so, then how can current ML models be open source? The source of a language model is the text it was trained on. Llama models are not open source (contrary to their claims), they are open weight.
You can find the entire Llama 3.0 pretraining set here: https://huggingface.co/datasets/HuggingFaceFW/fineweb 15T tokens, 45 terrabytes. Seems fairly open source to me.
Earlier quoted context omitted.
Feels a bit like you are splitting hair for the pleasure of semantic arguments to be honest. Yes there are no source in ML, so if we want to be pedantic it shouldn't be called open source. But what really matters in the open source movement is that we are able to take a program built by someone and modify it to do whatever we want with it, without having to ask someone for permission or get scrutinized or have to pay…
Calling that “open source” renders the word “source” meaningless. By your definition, I can release a binary executable freely and call it “open source” because you can modify it to do whatever you want. Model weights are like a binary that nobody has the source for. We need another term.
Here, modifying that model is not harder that doing regular ML, and I can redistribute.
Meta doesn’t have access to some magic higher level abstraction for that model that would make working with it easier that they did not release.
The sources in ML are the architecture the training and inference code and a paper describing the training procedure. It’s all there.
Earlier quoted context omitted.
I'm just a lowly outsider to the AI space, but calling these open source models seems kind of like calling a compiled binary open source. If you don't have a way to replicate what they did to create the model, it seems more like freeware than open source.
As an ML researcher, I agree. Meta doesn't include adequate information to replicate the models, and from the perspective of fundamental research, the interest that big tech companies have taken in this field has been a significant impediment to independent researchers, despite the fact that they are undeniably producing groundbreaking results in many respects, due to this fundamental lack of openness This should als…
Earlier quoted context omitted.
I'm just a lowly outsider to the AI space, but calling these open source models seems kind of like calling a compiled binary open source. If you don't have a way to replicate what they did to create the model, it seems more like freeware than open source.
As an ML researcher, I agree. Meta doesn't include adequate information to replicate the models, and from the perspective of fundamental research, the interest that big tech companies have taken in this field has been a significant impediment to independent researchers, despite the fact that they are undeniably producing groundbreaking results in many respects, due to this fundamental lack of openness This should als…
A) Release the data, and if it ends up causing a privacy scandal, at least you can actually call it open this time.
B) Neuter the dataset, and the model
All I ever see in these threads is a lot of whining and no viable alternative solutions (I’m fine with the idea of it being a hard problem, but when I see this attitude from “researchers” it makes me less optimistic about the future)
> and the credulity that otherwise smart people have been exhibiting in this space has been baffling to me
Remove the “otherwise” and you’re halfway to understanding your error.
Earlier quoted context omitted.
Makes me wonder why he's really doing this. Zuckerberg being Zuckerberg, it can't be out of any genuine sense of altruism. Probably just wants to crush all competitors before he monetizes the next generation of Meta AI.
Its certainly not altruism. Given that Facebook/Meta owns the largest user data collection systems, any advancement in AI ultimately strengthens their business model (which is still mostly collecting private user data, amassing large user datasets, and selling targeting ads). There is a demo video that shows a user wearing a Quest VR headset and asks the AI "what do you see" and it interprets everything around it. Th…
Earlier quoted context omitted.
Is forking really possible with an LLM or one the size of future Lama versions, have they even released the weights and everything? Maybe I am just negative about it because I feel Meta is the worst company ever invented and feel this will hurt society in the long run just like Facebook.
When you run `ollama pull llama3.1:70b`, which you can literally do right now (assuming ollama.com is installed and you're not afraid of the terminal), and it downloads a 40 gigabyte model, that is the weights ! I'd consider the ability to admit when even your most hated adversary is doing something right, a hallmark of acting smarter. Now, they haven't released the training data with the model weights. THAT plus the…
Hard disagree. So far every big important model is closed-source. Grok is sort-of the only exception, and it's not even that big compared to the (already old) GPT-4. I don't see open source being able to compete with the cutting-edge proprietary models. There's just not enough money. GPT-5 will take an estimated $1.2 billion to train. MS and OpenAI are already talking about building a $100 billion training data cente…
Where is the $1.2b number from?
HSBC estimates the training cost for GPT-5 between $1.7B and $2.5B.
Vlad Bastion Research estimates $1.25B - 2.25B.
Some people on HN estimate $10B:
> We’re releasing Llama 3.1 405B, the first frontier-level open source AI model, as well as new and improved Llama 3.1 70B and 8B models. Bravo! While I don't agree with Zuck's views and actions on many fronts, on this occasion I think he and the AI folks at Meta deserve our praise and gratitude. With this release, they have brought the cost of pretraining a frontier 400B+ parameter model to ZERO for pretty much ever…
Praising is good. Gratitude is a bit much. They got this big by selling user generated content and private info to the highest bidder. Often through questionable means. Also, the underdog always touts Open Source and standards, so it’s good to remain skeptical when/if tables turn.
Earlier quoted context omitted.
> If that ever stops being the case, you should fork rather than trust their judgment. Fork what? The secret sauce is in the training data and infrastructure. I don't think either of those is currently open.
I'm just a lowly outsider to the AI space, but calling these open source models seems kind of like calling a compiled binary open source. If you don't have a way to replicate what they did to create the model, it seems more like freeware than open source.
Isn't that a bit like arguing that a linux kernel driver isn't open source if I just give you a bunch of GPL-licensed source code that speaks to my device, but no documentation how my device works? If you take away the source code you have no way to recreate it. But so far that never caused anyone to call the code not open-source. The closest is the whole GPL3 Tivoization debate and that was very divisive.
The heart of the issue is that open source is kind of hard to define for anything that isn't software. As a proxy we could look at Stallman's free software definition. Free software shares a common history with open source and in most open source software is free/libre, and the other way around, so this might be a useful proxy.
So checking the four software freedoms:
- The freedom to run the program as you wish, for any purpose: For most purposes. There's that 700M user restriction, also Meta forbids breaking the law and requires you to follow their acceptable use policy.
- The freedom to study how the program works, and change it so it does your computing as you wish: yes. You can change it by fine tuning it, and the weights allow you to figure out how it works. At least as well as anyone knows how any large neural network works, but it's not like Meta is keeping something from you here
- The freedom to redistribute copies so you can help your neighbor: Allowed, no real asterisks
- The freedom to distribute copies of your modified versions to others: Yes
So is it Free Software™? Not really, but it is pretty close.
Earlier quoted context omitted.
You have proposed state ownership of all successful IP. That is a massive change and yet you have demonstrated zero understanding of the possible costs. Your claim that removing a profit motivation will increase investment is flat out wrong. Everything else crumbles from there.
No, I've proposed removing or reducing IP protections, not transferring them to the state. Allowing competitors to enter the market will obviously increase investment in competitors...
I do think there are some serious IP issues, as IP rules can be hijacked in the US, but that means you fix those problems, not blow up IP that was rightfully earned