Trinity large: An open 400B sparse MoE model
1–10 of 83 posts
Re: Trinity large: An open 400B sparse MoE model
#2I'm particularly excited to see a "true base" model to do research off of (https://huggingface.co/arcee-ai/Trinity-Large-TrueBase).
Re: Trinity large: An open 400B sparse MoE model
#3They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive
Re: Trinity large: An open 400B sparse MoE model
#4This is a wonderful release.
Re: Trinity large: An open 400B sparse MoE model
#5They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive
The price/scaling of training another same class model always seems to be dropping through the floor but training models which score much better seems to be hitting a brick wall.
E.g. gemini-3-pro tops the lmarena text chart today at 1488 vs 1346 for gpt-4o-2024-05-13. That's a win rate of 70% (where 50% is equal chance of winning) over 1.5 years. Meanwhile, even the open weights stuff OpenAI gave away last summer scores between the two.
The exception seems to be net new benchmarks/benchmark versions. These start out low and then either quickly get saturated or hit a similar wall after a while.
Re: Trinity large: An open 400B sparse MoE model
#6What exactly does "open" mean in this case? Is it weights and data or just weights?
Re: Trinity large: An open 400B sparse MoE model
#7Given that it's a 400B-parameter model, but it's a sparse MoE model with 13B active parameters per token, would it run well on an NVIDIA DGX Spark with 128 GB of unified RAM, or do you practically need to hold the full model in RAM even with sparse MoE?
Re: Trinity large: An open 400B sparse MoE model
#8The only thing I question is the use of Maverick in their comparison charts. That's like comparing a pile of rocks to an LLM.
Re: Trinity large: An open 400B sparse MoE model
#9What exactly does "open" mean in this case? Is it weights and data or just weights?
It's always open weights.
Re: Trinity large: An open 400B sparse MoE model
#10Given that it's a 400B-parameter model, but it's a sparse MoE model with 13B active parameters per token, would it run well on an NVIDIA DGX Spark with 128 GB of unified RAM, or do you practically need to hold the full model in RAM even with sparse MoE?
Even with MoE, holding the model in RAM while individual experts are evaluated in VRAM is a bit of a compromise. Experts can be swapped in and out of VRAM for each token. So RAM VRAM bandwidth becomes important. With a model larger than RAM, that bandwidth bottleneck gets pushed to the SSD interface. At least it's read-only, and not read-write, but even the fastest of SSDs will be significantly slower than RAM.
That said, there are folks out there doing it. https://github.com/lyogavin/airllm is one example.