141–150 of 264 posts
Deepseek seems to be one of the few LLMs that run on a iPod Touch because of the older version of ios.
Hey! You! You can't just say that and not explain. Come back.
Well that didn't take long, available from 7 providers through openrouter. https://openrouter.ai/deepseek/deepseek-r1-0528/providers May 28th update to the original DeepSeek R1 Performance on par with OpenAI o1, but open-sourced and with fully open reasoning tokens. It's 671B parameters in size, with 37B active in an inference pass. Fully open-source model.
It's. not. open. source! https://www.downloadableisnotopensource.org/
The companies that create these models cant answer that question! Models get jailbroken all the time to ignore alignment instructions. The robust refusal logic normally sits on top of the model, ie looking at the responses and flagging anything that they don't want to show to users.
The best tool we have for understanding if a model is refusing to answer a problem or actually doesn't know is mechanistic interp, which you only need the weights for.
This whole debate is weird, even with traditional open source code you cant tell the intent of a programmer, what sources they used to write that code etc.
Earlier quoted context omitted.
No sign of what source material it was trained on though right? So open weight rather than reproducible from source. I remember there's a project "Open R1" that last I checked was working on gathering their own list of training material, looks active but not sure how far along they've gotten: https://github.com/huggingface/open-r1
> No sign of what source material it was trained on though right? out of curiosity, does anyone do anything "useful" with that knowledge? it's not like people can just randomly train models..
Today we introduce OLMoTrace, a one-of-a-kind feature in the Ai2 Playground that lets you trace the outputs of language models back to their full, multi-trillion-token training data in real time. OLMoTrace is a manifestation of Ai2’s commitment to an open ecosystem – open models, open data, and beyond.
Earlier quoted context omitted.
It's. not. open. source! https://www.downloadableisnotopensource.org/
Open source is a crazy new beast in the AI/ML world. We have numerous artifacts to reason about: - The model code - The training code - The fine tuning code - The inference code - The raw training data - The processed training data (which might vary across various stages of pre-training and potentially fine-tuning!) - The resultant weights - The inference outputs (which also need a license) - The research papers (hop…
Earlier quoted context omitted.
Open source is a crazy new beast in the AI/ML world. We have numerous artifacts to reason about: - The model code - The training code - The fine tuning code - The inference code - The raw training data - The processed training data (which might vary across various stages of pre-training and potentially fine-tuning!) - The resultant weights - The inference outputs (which also need a license) - The research papers (hop…
I think you‘re trying to make it look more complex than it is. Put the amount of data next to every entry in that list of yours.
If you think the data story isn't a complicated beast, then consider:
If you wanted an "open" dataset, would you want it before or after it was processed? There are a lot of cleaning, categorizing, feature extraction steps. The data typically undergoes a lot of analysis, extra annotation, bucketing, and transformation.
If the pre-train was done in stages, and the training process was complicated, how much hand-holding do you need to replicate that process?
Do you need all of the scripts to assist with these processes? All of the infra and MLOps pieces? There's a lot of infrastructure to just move the data around and poke it.
Where are you going to host those terabytes or petabytes of data? Who is going to download it? How often? Do you expect it to be downloaded as frequently as the Linux kernel sources?
Did you scrub it of PII? Are you sure?
And to clarify, we're not even talking about trained models at this point.
Earlier quoted context omitted.
No sign of what source material it was trained on though right? So open weight rather than reproducible from source. I remember there's a project "Open R1" that last I checked was working on gathering their own list of training material, looks active but not sure how far along they've gotten: https://github.com/huggingface/open-r1
Isn't it basically not possible for the input data set list to be listed? It's an open secret all these labs are using immense amounts of copyrighted material. There's a few efforts at full open data / open weight / open code models, but none of them have gotten to leading-edge performance.
Earlier quoted context omitted.
No sign of what source material it was trained on though right? So open weight rather than reproducible from source. I remember there's a project "Open R1" that last I checked was working on gathering their own list of training material, looks active but not sure how far along they've gotten: https://github.com/huggingface/open-r1
Isn't it basically not possible for the input data set list to be listed? It's an open secret all these labs are using immense amounts of copyrighted material. There's a few efforts at full open data / open weight / open code models, but none of them have gotten to leading-edge performance.
Earlier quoted context omitted.
Groq has a weak selection of models, which is frustrating because their inference speed is insane. I get it though, selection + optimization = performance.
the only reason they are fast is because the models they host are severely quantized so i've heard.
Earlier quoted context omitted.
Isn't it basically not possible for the input data set list to be listed? It's an open secret all these labs are using immense amounts of copyrighted material. There's a few efforts at full open data / open weight / open code models, but none of them have gotten to leading-edge performance.
“Not possible” = “a business-destroying level of honesty”?