81–90 of 168 posts
Meta is going to ruin open ais moat on purpose. Great business strategy and good for everyone but metas competitors
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
It's not open source, it's freeware or something like that. Weights aren't the source code of LLMs, they're the binaries.
Still waiting for the 'Meta is dying' and 'Fire Mark Zuckerberg' calls from last year. A year later, where are they now?
Earlier quoted context omitted.
Maybe there is no source code? I imagine an LLM is like output of the following process. There's a huge room full of programmers that can directly edit machine code. You give them a random binary, which they then hack on for a while and publish the result. You then inspect it and tell them it isn't quite optimal in some way and ask them for a new version. Iterate on this process a bazillion times. At the end you get…
Source code is the preferred form for development. In your scenario, despite the unrealistic coding process, the machine code is the source code, because that's what everyone is working on. In the development of LLM, the weights is in no way the preferred form of development. Programmers don't work on weights. They work on data, infrastructure, the model, the code for training, etc. The point of machine learning is n…
You release your weights, others can build on top of that, fine tune it in different ways, produce new weights they can share with others. Seems very OSS-y.
I feel like there is some semantic nitpicky point being made here that is completely going over my head.
Earlier quoted context omitted.
> It's not like Meta can remove these books from the training set without retraining from scratch (or at least the last checkpoint before they were used). They probably can: https://github.com/zjunlp/EasyEdit > I wonder if this is going to cause issues down the road. There are some popular Stable Diffusion models, being run in small businesses, that I am certain have CSAM in them because they have a particular 4chan…
> They probably can: No, actually they probably can’t. There is no verifiable way to remove the data from the model apart from completely removing all instances of information from the training data. The project you linked only describes a selective finetuning approach.
... But yeah, fundamentally the only way to throw out the books is to throw out the weights.
Earlier quoted context omitted.
> It's not like Meta can remove these books from the training set without retraining from scratch (or at least the last checkpoint before they were used). They probably can: https://github.com/zjunlp/EasyEdit > I wonder if this is going to cause issues down the road. There are some popular Stable Diffusion models, being run in small businesses, that I am certain have CSAM in them because they have a particular 4chan…
> They probably can: No, actually they probably can’t. There is no verifiable way to remove the data from the model apart from completely removing all instances of information from the training data. The project you linked only describes a selective finetuning approach.
From the recent story about the Sarah Silverman lawsuit: The complaint lays out in steps why the plaintiffs believe the datasets have illicit origins — in a Meta paper detailing LLaMA, the company points to sources for its training datasets, one of which is called ThePile, which was assembled by a company called EleutherAI. ThePile, the complaint points out, was described in an EleutherAI paper as being put together…
Earlier quoted context omitted.
Move fast and break the law.
Copyright laws should be amended to allow this scenario. If I read a book and write about it in a blog, it is considered review. Why shouldn’t we allow companies to do the same to train their models? Overall it will benefit society more than it hurts some rich authors.
From the recent story about the Sarah Silverman lawsuit: The complaint lays out in steps why the plaintiffs believe the datasets have illicit origins — in a Meta paper detailing LLaMA, the company points to sources for its training datasets, one of which is called ThePile, which was assembled by a company called EleutherAI. ThePile, the complaint points out, was described in an EleutherAI paper as being put together…
My guess is still the latter because that's what I've heard the rumors about, but this article is pretty unclear on this fact.