Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Trinity large: An open 400B sparse MoE model

arcee.ai

Post reply on HN
Print view View original

11–20 of 83 posts

Re: Trinity large: An open 400B sparse MoE model

#11

Post by antirez »

mwcampbell wrote: post #7

Given that it's a 400B-parameter model, but it's a sparse MoE model with 13B active parameters per token, would it run well on an NVIDIA DGX Spark with 128 GB of unified RAM, or do you practically need to hold the full model in RAM even with sparse MoE?

Can run with mmap() but it is slower. 4-bit quantized there is a decent ratio between the model size and the RAM, with a fast SSD one could try to see how it works. However when a model is 4-bit quantized there is often the doubt that it is not better than an 8-bit quantized model of 200B parameters, it depends on the model, on the use case, ... Unfortunately the street for local inference of SOTA model is being stopped by the RAM prices and the GPU request of the companies, leaving us with little. Probably today the best bet is to buy Mac Studio systems and then run distributed inference (MLX supports this for instance), or a 512 GB Mac Studio M4 that costs, like 13k$.

Re: Trinity large: An open 400B sparse MoE model

#15

Post by linolevan »

jacquesm wrote: post #13
jetpackjoe wrote: post #12

Earlier quoted context omitted.

It's never open data

Well, it is, it's your data to begin with after all but admitting that would create some problems.

This model is sort of interesting since it seems to be using a lot of synthetic training data – but your point stands

Re: Trinity large: An open 400B sparse MoE model

#16

Post by notpublic »

antirez wrote: post #11
mwcampbell wrote: post #7

Given that it's a 400B-parameter model, but it's a sparse MoE model with 13B active parameters per token, would it run well on an NVIDIA DGX Spark with 128 GB of unified RAM, or do you practically need to hold the full model in RAM even with sparse MoE?

Can run with mmap() but it is slower. 4-bit quantized there is a decent ratio between the model size and the RAM, with a fast SSD one could try to see how it works. However when a model is 4-bit quantized there is often the doubt that it is not better than an 8-bit quantized model of 200B parameters, it depends on the model, on the use case, ... Unfortunately the street for local inference of SOTA model is being stop…

Talking about RAM prices, you can still get a framework Max+ 395 with 128GB RAM for ~$2,459 USD. They have not increased the price for it yet.

https://frame.work/products/desktop-diy-amd-aimax300/configu...

Re: Trinity large: An open 400B sparse MoE model

#18

Post by QuadmasterXLII »

Alifatisk wrote: post #14

What did they do to make the loss drop so much in phase 3? Also, why are they comparing with Llama 4 Maverick? Wasn’t it a flop?

you can’t directly compare losses because they changed the data distribution for each phase ( I think. 100% guaranteed they change the data distribution after the 10 trillion token mark, that’s when they start adding in instruction following data, but I don’t know for sure if the other phase changes also include data distribution changes.)

Re: Trinity large: An open 400B sparse MoE model

#19

Post by Scipio_Afri »

notpublic wrote: post #16
antirez wrote: post #11

Earlier quoted context omitted.

Can run with mmap() but it is slower. 4-bit quantized there is a decent ratio between the model size and the RAM, with a fast SSD one could try to see how it works. However when a model is 4-bit quantized there is often the doubt that it is not better than an 8-bit quantized model of 200B parameters, it depends on the model, on the use case, ... Unfortunately the street for local inference of SOTA model is being stop…

Talking about RAM prices, you can still get a framework Max+ 395 with 128GB RAM for ~$2,459 USD. They have not increased the price for it yet. https://frame.work/products/desktop-diy-amd-aimax300/configu...

Pretty sure those use to be $1999 ... but not entirely sure

Re: Trinity large: An open 400B sparse MoE model

#20

Post by gwern »

zamadatix wrote: post #5
mynti wrote: post #3

They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive

The price/scaling of training another same class model always seems to be dropping through the floor but training models which score much better seems to be hitting a brick wall. E.g. gemini-3-pro tops the lmarena text chart today at 1488 vs 1346 for gpt-4o-2024-05-13. That's a win rate of 70% (where 50% is equal chance of winning) over 1.5 years. Meanwhile, even the open weights stuff OpenAI gave away last summer sc…

> E.g. gemini-3-pro tops the lmarena text chart today at 1488 vs 1346 for gpt-4o-2024-05-13. That's a win rate of 70% (where 50% is equal chance of winning) over 1.5 years. Meanwhile, even the open weights stuff OpenAI gave away last summer scores between the two.

Why do you care about LM Arena? It has so many problems, and the fact that no one would suggest using GPT-4o for doing math or coding right now, or much of anything, should tell you that a 'win rate of 70%' does not mean whatever it looks like it means. (Does GPT-4o solve roughly as many Erdos questions as gemini-3-pro...? Can you write roughly as good poetry?)

Post reply on HN