I'm particularly excited to see a "true base" model to do research off of ( https://huggingface.co/arcee-ai/Trinity-Large-TrueBase ).
I'd love to "chat" to that model see how it behaves
Trinity large: An open 400B sparse MoE model
41–50 of 83 posts
Re: Trinity large: An open 400B sparse MoE model
#42Earlier quoted context omitted.
I've always found LiveBench a bit confusing to try to compare over time as the dataset isn't meant to be compared over time. It also currently claims GPT-5 Mini High from last summer is within ~15% of Claude 4.5 Opus Thinking High Effort in the average, but I'll wait with bated breath for the millions of amazing apps which couldn't be coded before to start showing up (or, more likely, be told in 6 months how these 2…
> I'll wait with bated breath for the millions of amazing apps which couldn't be coded before to start showing up Appstore releases were roughly linear until July 25 and are up 60% since then: https://www.coatue.com/c/takes/chart-of-the-day-2026-01-22
One of the best surgically executed nukes on HN in my 16 years here.
Re: Trinity large: An open 400B sparse MoE model
#43Earlier quoted context omitted.
Frankly, this reads as a lot of words that amount to an excuse for using only LMArena, and the rationale is quite clear: it’s for an unrelated argument that isn’t going to ring true to people, especially an audience of programmers who just spent the last year watching the AI go from being able to make coherent file edits to multi hour work. LMArena is, de facto, a sycophancy and Markdown usage detector. Two others yo…
I've always found LiveBench a bit confusing to try to compare over time as the dataset isn't meant to be compared over time. It also currently claims GPT-5 Mini High from last summer is within ~15% of Claude 4.5 Opus Thinking High Effort in the average, but I'll wait with bated breath for the millions of amazing apps which couldn't be coded before to start showing up (or, more likely, be told in 6 months how these 2…
See peer reply re: yes, your self-chosen benchmark has been reached.
Generally, I've learned to warn myself off of a take when I start writing emotionally charged stuff like [1]. Without any prompting (who mentioned apps? and why would you without checking?), also, when reading minds, and assigning weak arguments, now and in my imagination of the future. [2]
At the very least, [2] is a signal to let the keyboard have a rest, and ideally my mind.
Bailey: > "If [there were] new LLMs...consistently solving Erdos problems at rapidly increasing rates then they'd be showing...that"
Motte: > "I can['t] pop into ChatGPT and pop out Erdos proofs regularly"
No less than Terence Tao, a month ago, pointing out your bailey was newly happening with the latest generation: https://mathstodon.xyz/@tao/115788262274999408. Not sure how you only saw one Erdos problem.
[1] "I'll wait with bated breath for the millions of amazing apps which couldn't be coded before to start showing up"
[2] "...or, more likely, be told in 6 months how these 2 benchmarks weren't the ones that should matter either"
Re: Trinity large: An open 400B sparse MoE model
#44They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive
I'll straight up accuse them of on purpose muddying the waters. To get to the point of executing a successful training run like that, you have to count every failed experiment and experiment that gets you to the final training run. They spent well over 100 Million to train this model by that definition, and all definitions which don't include the failed runs up to the successful one at the end are at best disingenuou…
> No Gemini was not "entirely trained on TPUs". They did hundreds of experiments on GPUs to get to the final training run done entirely on TPUs. GCP literally has millions of GPUs and you bet your ass that the gemini team has access to them and uses them daily.
You are wrong. Gemini was definitely trained entirely on TPU. Of course your point of "you need to count failed experiments, too". Is correct. But you seem to have misconceptions around how deepmind operates and what infra it possess. Deepmind (or barely any of Google internal stuff) runs on Borg, an internal cloud system, which is completely separate (and different) from gcp. Deepmind does not have access to any meaningful gcp resources. And Borg barely has any GPUs. At the time I left deepmind, the amount of tpu compute available was probably 1000x to 10000x larger than the amount of gpu compute. You would never even think of seriously using GPUs for neural net training, it's too limited (in terms of available compute) and expensive (in terms of internal resource allocation units), and frankly less well supported by internal tooling than tpu. Even for small, short experiments, you would always use TPUs.
Re: Trinity large: An open 400B sparse MoE model
#45Given that it's a 400B-parameter model, but it's a sparse MoE model with 13B active parameters per token, would it run well on an NVIDIA DGX Spark with 128 GB of unified RAM, or do you practically need to hold the full model in RAM even with sparse MoE?
Even with MoE, holding the model in RAM while individual experts are evaluated in VRAM is a bit of a compromise. Experts can be swapped in and out of VRAM for each token. So RAM VRAM bandwidth becomes important. With a model larger than RAM, that bandwidth bottleneck gets pushed to the SSD interface. At least it's read-only, and not read-write, but even the fastest of SSDs will be significantly slower than RAM. That…
With a non-sequential generative approach perhaps the RAM cache misses could be grouped together and swapped on a when available/when needed prioritized bases.
Re: Trinity large: An open 400B sparse MoE model
#46> We optimize for performance per parameter and release weights under Apache-2.0 How do they plan to monetize?
I'm guessing by selling fine-tuning, consulting on hosting, and other services? They also seem to be offering their own inference service with their model, obviously as an open weight model that will be commoditized but I'm sure there are some people who'd prefer to buy from the originating lab. But yeah, when you're offering open weights models, your customers are going to be people who want to self-host, fine tune, etc, so they might be offering services for that.
Re: Trinity large: An open 400B sparse MoE model
#47Earlier quoted context omitted.
I'll straight up accuse them of on purpose muddying the waters. To get to the point of executing a successful training run like that, you have to count every failed experiment and experiment that gets you to the final training run. They spent well over 100 Million to train this model by that definition, and all definitions which don't include the failed runs up to the successful one at the end are at best disingenuou…
> No Gemini was not "entirely trained on TPUs". They did hundreds of experiments on GPUs to get to the final training run done entirely on TPUs. GCP literally has millions of GPUs and you bet your ass that the gemini team has access to them and uses them daily. You are wrong. Gemini was definitely trained entirely on TPU. Of course your point of "you need to count failed experiments, too". Is correct. But you seem to…
At least blessed teams we used GPUs when we were allowed, else CPUs. TPUs were basically banned in YT since they were reserved for higher priority purposes. Gemini was almost certainly trained with one, but I guarantee an ungodly amount of compute has gone into training neural nets with CPUs and GPUs.
Re: Trinity large: An open 400B sparse MoE model
#48Given that it's a 400B-parameter model, but it's a sparse MoE model with 13B active parameters per token, would it run well on an NVIDIA DGX Spark with 128 GB of unified RAM, or do you practically need to hold the full model in RAM even with sparse MoE?
Can run with mmap() but it is slower. 4-bit quantized there is a decent ratio between the model size and the RAM, with a fast SSD one could try to see how it works. However when a model is 4-bit quantized there is often the doubt that it is not better than an 8-bit quantized model of 200B parameters, it depends on the model, on the use case, ... Unfortunately the street for local inference of SOTA model is being stop…
I think 512 GB Mac Studio was M3 Ultra.
Anyways, isn't a new Mac Studio due in a few months? It should be significantly faster as well.
I just hope RAM prices don't ruin this...
Re: Trinity large: An open 400B sparse MoE model
#49They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive
Why even do such thing if there is free Google, chatgpt and dozen more models? Waste of money towards ultimate goal: global loss of jobs and destroying earth.