Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Trinity large: An open 400B sparse MoE model

arcee.ai

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Trinity large: An open 400B sparse MoE model

#51

Post by tgrowazay »

mynti wrote: post #3

They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive

> 2048 Nvidia B300 GPU

With average price of $6/hour that is $12,288/hour for whole cluster.

Times 33 days times 24 hours it comes out to be $9.7MM , assuming no discounts.

That leaves $10.3MM/6 months for salaries, which is 103 employees at $200k/year or 51 employee at $400k/year.

Re: Trinity large: An open 400B sparse MoE model

#52

Post by YetAnotherNick »

zamadatix wrote: post #5
mynti wrote: post #3

They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive

The price/scaling of training another same class model always seems to be dropping through the floor but training models which score much better seems to be hitting a brick wall. E.g. gemini-3-pro tops the lmarena text chart today at 1488 vs 1346 for gpt-4o-2024-05-13. That's a win rate of 70% (where 50% is equal chance of winning) over 1.5 years. Meanwhile, even the open weights stuff OpenAI gave away last summer sc…

> The exception seems to be net new benchmarks/benchmark versions.

How is this an exception? If a genius and kindergarden student takes a test to add two single digit numbers how is that result any relevant? Even though adding single digit number is in the class of possible test.

We can only look at non saturated test.

Re: Trinity large: An open 400B sparse MoE model

#53

Post by YetAnotherNick »

tgrowazay wrote: post #51
mynti wrote: post #3

They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive

> 2048 Nvidia B300 GPU With average price of $6/hour that is $12,288/hour for whole cluster. Times 33 days times 24 hours it comes out to be $9.7MM , assuming no discounts. That leaves $10.3MM/6 months for salaries, which is 103 employees at $200k/year or 51 employee at $400k/year.

It would likely be something like $4.5/hour for this big cluster.

[1]: https://verda.com/products#B300

Re: Trinity large: An open 400B sparse MoE model

#54

Post by YetAnotherNick »

querez wrote: post #44
Der_Einzige wrote: post #40

Earlier quoted context omitted.

I'll straight up accuse them of on purpose muddying the waters. To get to the point of executing a successful training run like that, you have to count every failed experiment and experiment that gets you to the final training run. They spent well over 100 Million to train this model by that definition, and all definitions which don't include the failed runs up to the successful one at the end are at best disingenuou…

> No Gemini was not "entirely trained on TPUs". They did hundreds of experiments on GPUs to get to the final training run done entirely on TPUs. GCP literally has millions of GPUs and you bet your ass that the gemini team has access to them and uses them daily. You are wrong. Gemini was definitely trained entirely on TPU. Of course your point of "you need to count failed experiments, too". Is correct. But you seem to…

Using TPU has the same opportunity cost as GPU. Just because they built something doesn't mean it's cheaper. If it is they can rent it cheaper to save money on paying billions of dollars to Nvidia.

A big segment of the market just uses GPU/TPU to train LLMs, so they don't exactly need flexibility if some tool is well supported.

Re: Trinity large: An open 400B sparse MoE model

#55

Post by dajonker »

nl wrote: post #38
zamadatix wrote: post #28

Earlier quoted context omitted.

It'd certainly be odd if people were recommending old LLMs which score worse, even if marginally. That said, 4o is really a lot more usable than you're making it out to be. The particular benchmark in the example is fungible but you have to pick something to make a representative example. No matter which you pick someone always has a reason "oh, it's not THAT benchmark you should look at". The benchmarks from the cha…

Progress has not become linear. We've just hit the limits of what we can measure and explain easily. One year ago coding agents could barely do decent auto-complete. Now they can write whole applications. That's much more difficult to show than an ELO score based on how people like emjois and bold text in their chat responses. Don't forget Llama4 led Lmarena and turned out to be very weak.

You are equally understating past performance as you are overstating current performance.

One year ago I already ran qwen2.5-coder 7B locally for pretty decent autocomplete. And I still use it today as I haven't found anything better, having tried plenty of alternatives.

Today I let LLM agents write probably 60-80% of the code, but I frequently have to steer and correct it and that final 20% still takes 80% of the time.

Re: Trinity large: An open 400B sparse MoE model

#56

Post by NitpickLawyer »

LoganDark wrote: post #36

According to the article, nearly 50% of the dataset is synthetic (8T out of 17T tokens). I don't know what constitutes "a breadth of state-of-the-art rephrasing approaches", but I lack some confidence in models trained on LLM output, so I hope it wasn't that.

> but I lack some confidence in models trained on LLM output, so I hope it wasn't that.

That's misguided. Models have been trained on synthetic data for ~2+ years already. The "model collapse" myth is based on a very poor paper that got waaaay more attention than it deserved (because negativity sells, I guess). In practice every lab out there is doing this, because it works.

Re: Trinity large: An open 400B sparse MoE model

#57

Post by Zababa »

zamadatix wrote: post #5
mynti wrote: post #3

They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive

The price/scaling of training another same class model always seems to be dropping through the floor but training models which score much better seems to be hitting a brick wall. E.g. gemini-3-pro tops the lmarena text chart today at 1488 vs 1346 for gpt-4o-2024-05-13. That's a win rate of 70% (where 50% is equal chance of winning) over 1.5 years. Meanwhile, even the open weights stuff OpenAI gave away last summer sc…

>E.g. gemini-3-pro tops the lmarena text chart today at 1488 vs 1346 for gpt-4o-2024-05-13. That's a win rate of 70% (where 50% is equal chance of winning) over 1.5 years. Meanwhile, even the open weights stuff OpenAI gave away last summer scores between the two.

I think in that specific case that says more about LMArena than about the newer models. Remember that GPT 4o was so specifically loved by people that when GPT 5 replaced there was lots of backlash against OpenAI.

One of the popular benchmarks right now is METR which shows some real improvement with newer models, like Opus 4.5. Another way of getting data is anecdotes, lots of people are really impressed with Opus 4.5 and Codex 5.2 (but they're hard distangle from people getting better with those tools, the scaffolding (Claude code, Codex) getting better, and lots of other stuff). SWEBench is still not saturated (less than 75% I think).

Re: Trinity large: An open 400B sparse MoE model

#58

Post by Zababa »

Der_Einzige wrote: post #40
mynti wrote: post #3

They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive

I'll straight up accuse them of on purpose muddying the waters. To get to the point of executing a successful training run like that, you have to count every failed experiment and experiment that gets you to the final training run. They spent well over 100 Million to train this model by that definition, and all definitions which don't include the failed runs up to the successful one at the end are at best disingenuou…

>To get to the point of executing a successful training run like that, you have to count every failed experiment and experiment that gets you to the final training run.

I get the sentiment, but then, do you count all the other experiments that were done by that company before specifically trying to train this model? All the experiments done by people in that company at other companies? Since they rely on that experience to train models.

You could say "count everything that has been done since the last model release", but then for the same amount of effort/GPU, if you release 3 models does that divide each model cost by 3?

Genuinely curious in how you think about this, I think saying "the model cost is the final training run" is fine as it seems standard ever since DeepSeek V3, but I'd be interested if you have alternatives. Possibly "actually don't even talk about model cost as it will always be misleading and you can never really spend the same amount of money to get the same model"?

Re: Trinity large: An open 400B sparse MoE model

#59

Post by Grimblewald »

hahahahhaah wrote: post #41
linolevan wrote: post #2

I'm particularly excited to see a "true base" model to do research off of ( https://huggingface.co/arcee-ai/Trinity-Large-TrueBase ).

I'd love to "chat" to that model see how it behaves

I highly recommend. As a tip, you can quite easily get into a chat like state by simply using in context learning. Have a few turns of conversation pre-written and generate from that. It'll continue the conversation (for both parties) so you just stop it from generating when it starts generating on your behalf.

That said, it's useful for so much more beyond. Outline the premise of a Book, then "what follows is that book\n #Chapter 1:" and watch it rip. Base models are my preferred way of using LLM's by a long margin.

Re: Trinity large: An open 400B sparse MoE model

#60

Post by radarsat1 »

timschmidt wrote: post #10
mwcampbell wrote: post #7

Given that it's a 400B-parameter model, but it's a sparse MoE model with 13B active parameters per token, would it run well on an NVIDIA DGX Spark with 128 GB of unified RAM, or do you practically need to hold the full model in RAM even with sparse MoE?

Even with MoE, holding the model in RAM while individual experts are evaluated in VRAM is a bit of a compromise. Experts can be swapped in and out of VRAM for each token. So RAM VRAM bandwidth becomes important. With a model larger than RAM, that bandwidth bottleneck gets pushed to the SSD interface. At least it's read-only, and not read-write, but even the fastest of SSDs will be significantly slower than RAM. That…

> Experts can be swapped in and out of VRAM for each token.

I've often wondered how much it happens in practice. What does the per-token distribution of expert selection actually look like during inference? For example does it act like uniform random variable, or does it stick with the same 2 or 3 experts for 10 tokens in a row? I haven't been able to find much info on this.

Obviously it depends on what model you are talking about, so some kind of survey would be interesting. I'm sure this must but something that the big inference labs are knowledgeable about.

Although, I guess if you are batching things, then even if a subset of experts is selected for a single query, maybe over the batch it appears completely random, that would destroy any efficiency gains. Perhaps it's possible to intelligently batch queries that are "similar" somehow? It's quite an interesting research problem when you think about it.

Come to think of it, how does it work then for the "prompt ingestion" stage, where it likely runs all experts in parallel to generate the KV cache? I guess that would destroy any efficiency gains due to MoE too, so the prompt ingestion and AR generation stages will have quite different execution profiles.

Post reply on HN