171–180 of 264 posts
Earlier quoted context omitted.
You can also reverse engineer and modify closed source programs (see mods for games). Weights are like compiled version of source data.
Not legally. That's the difference.
Earlier quoted context omitted.
The weights seem much more like a binary to me, the training pipeline the compiler, and the training dataset the source.
Come here to write this - perfect analogy!
Earlier quoted context omitted.
Come here to write this - perfect analogy!
It's very imperfect analogy though these things can't be rebuilt "from scratch" like a program, the training process doesn't seem to be replicable anyway. Nonetheless, full data disclosure is necessary, according to the result of the years-long consultation led by the Open Source Initiative https://opensource.org/ai
The training process is fully deterministic. It's just an algorithm. Feed the same data in and you'll get the same weights out.
If you're speaking about the computational cost, it used to be that way for compilers too. Give it 20 years and you'll be able to train one of today's models on your phone.
Earlier quoted context omitted.
Honest question, how do you know this is a big improvement? Are there any benchmarks anywhere?
There will be a video from FireShip if its a big one. /s
Earlier quoted context omitted.
I don't think people make the distinction like that. The open source vs non open source distinction boils down to, usually, can you use it for commercial use. what you're saying is just that it's non reproducible, which is a completely valid but separate issue
But where's the source? I just see a binary blob, what makes it open source?
Earlier quoted context omitted.
Benchmarks seem like a fools errand at this point; overly tuning models just to specific test already published tests, rather than focusing on making them generalize. Hugging face has a leader board and it seems dominated by models that are finetunings of various common open source models, yet don't seem be broader used: https://huggingface.co/open-llm-leaderboard
There are quite a few benchmarks for which that's not the case: - live benchmarks (livebench, livecodebench, matharena, SWE-rebench, etc) - benchmarks that do not have a fixed structure, like games or human feedback benches (balrog, videogamebench, arena) - (to some extent) benchmark without existing/published answers (putnambench, frontiermath). You could argue that someone could hire people to solve those or pay of…
Of course, some benchmarks are still valid and will remain valid. Ie. we can make the models play chess against each other and score them on how well they do. But those benchmarks are in general fairly narrow. They don't really measure the "broader" intelligence we are after. And often, LLMs perform worse than specialized models. Ie. I don't think there is any LLM out there that can beat a traditional chess program (surely not using the same computing power).
What is really bad are the QA benchmarks which leak over time into the training data of the models. And sometimes, one can suspect even big labs have an economic incentive in scoring well on popular benchmarks which cause them to manipulate the models way beyond what is reasonable.
And taking a bunch of flawed benchmarks and combining them in indexes, saying this model is 2% better than that model is just completely meaningless but of course fun and draws a lot of attention.
So, yes, we are kind of left with vibe checks, but in theory, we could do more; take a bunch of models, double-blind, and have a big enough, representative group of human evaluators score them against each other on meaningful subjects.
Of course, done right, that would be really expensive. And those sponsoring might not like the result.
Earlier quoted context omitted.
It's very imperfect analogy though these things can't be rebuilt "from scratch" like a program, the training process doesn't seem to be replicable anyway. Nonetheless, full data disclosure is necessary, according to the result of the years-long consultation led by the Open Source Initiative https://opensource.org/ai
> the training process doesn't seem to be replicable anyway The training process is fully deterministic. It's just an algorithm. Feed the same data in and you'll get the same weights out. If you're speaking about the computational cost, it used to be that way for compilers too. Give it 20 years and you'll be able to train one of today's models on your phone.
Earlier quoted context omitted.
> the training process doesn't seem to be replicable anyway The training process is fully deterministic. It's just an algorithm. Feed the same data in and you'll get the same weights out. If you're speaking about the computational cost, it used to be that way for compilers too. Give it 20 years and you'll be able to train one of today's models on your phone.
Can you point at the research that says that the training process of a LLM at least the size of OLMo or Pythia is deterministic?
Earlier quoted context omitted.
Isn't it basically not possible for the input data set list to be listed? It's an open secret all these labs are using immense amounts of copyrighted material. There's a few efforts at full open data / open weight / open code models, but none of them have gotten to leading-edge performance.
“Not possible” = “a business-destroying level of honesty”?
Earlier quoted context omitted.
It's very imperfect analogy though these things can't be rebuilt "from scratch" like a program, the training process doesn't seem to be replicable anyway. Nonetheless, full data disclosure is necessary, according to the result of the years-long consultation led by the Open Source Initiative https://opensource.org/ai
> the training process doesn't seem to be replicable anyway The training process is fully deterministic. It's just an algorithm. Feed the same data in and you'll get the same weights out. If you're speaking about the computational cost, it used to be that way for compilers too. Give it 20 years and you'll be able to train one of today's models on your phone.
Training is reproducible only if, besides the pipeline and data, you also start from the same random weights.