Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

github.com

Post reply on HN
Print view View original

121–130 of 261 posts

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#121

Post by rafaquintanilha »

I have no affiliation with them but here's what I think happened:

1. They claim the official model is based on Qwen 397B. It's likely they didn't disclose Nex Pro at all because Nex itself is based on the same base model (not saying they shouldn't).

2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all.

3. It's important to notice they didn't advertise the model besides posting it on Reddit 2 days ago. It became viral organically, over the weekend, and during Brazil's World Cup debut (Brazilians will understand). Of course the mayor of Rio took the opportunity to capitalize over the free coverage, but that wasn't done in conjunction with the researchers.

4. I don't see why they would disclose Qwen 397B as base and mention the SwiReasoning paper but not mention Nex if all they did was to merge both models.

5. In any case, what they are claiming is easily verifiable once (if) they upload the right model.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#122

Post by themafia »

Scroll_Swe wrote: post #119
themafia wrote: post #117

Earlier quoted context omitted.

Two spoiled rich kids arguing over who's morality is the least worst. That this moment is held up as some great exchange in business is annoying. That our regulatory agencies are perennially sleep at the switch and allow this nonsense to keep happening is extremely frustrating.

[flagged]

Let me guess, when confronted with uncomfortable information that requires you to think longer than you are used to, you devolve to false dichotomies into defend your ego?

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#124

Post by throwa356262 »

rafaquintanilha wrote: post #121

I have no affiliation with them but here's what I think happened: 1. They claim the official model is based on Qwen 397B. It's likely they didn't disclose Nex Pro at all because Nex itself is based on the same base model (not saying they shouldn't). 2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all. 3. It's…

Regarding #2

https://news.ycombinator.com/item?id=48529544

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#125

Post by oofbey »

x312 wrote: post #96
hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

This works because Nex itself is a finetune of Qwen3.5 ( https://huggingface.co/nex-agi/Nex-N2-Pro ). It's merging Qwen3.5 with a Qwen3.5 finetune. I don't believe this would work on two LLMs that have different pretraining. Even if it did you would need two LLMs that have exact same internal activation shapes, dimensions, expert counts, token vocabulary, realistically it would never happen outside of finetunes or ac…

Correct. We used to think that because NN optimization is non-convex there are all these local minima. Now we know that once you get past the very early parts of training from random init, the loss surface is fairly smooth, and not really convex, but close enough in a bunch of ways - linear combinations of trained models are pretty much always valid combinations. You can think of fine tunings as deltas on the original model which can be summed together successfully. I think this paper first showed that to me: https://arxiv.org/pdf/1802.10026 which was 8 years ago now.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#126

Post by jdiff »

valleyer wrote: post #107
jdiff wrote: post #103

Earlier quoted context omitted.

Without the system prompt, asking its name results in it responding with the name of the model they're ripping from. That would certainly draw your eyes to the right places.

Why is this? Do labs reinforce the model name during training? I was under the impression that this sort of "self-knowledge" always came from the system prompt, but I guess not...

Yes. In this case, during fine tuning. Other blurbs are also baked in during fine tuning that are perfectly reproducible from the Nex model. The details inside the linked issue are quite accessible.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#127

Post by vvpan »

root-parent wrote: post #19
ekjhgkejhgk wrote: post #15

One funny thing about incompetence is that they don't have the competence to know that their incompetence is straightforward to verify by a competent person.

You just described every single vibe coder...

I think that's unfair to "vibe coding". If anybody explicitly claims to vibe coding something than they are admitting to low supervision of the code. And on the contrary you can also AI-produce code that you have supervised highly. I suppose there are people who both AI their code and push it as bespoke but I, for one, have not met such a person at our outside of work.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#128

Post by ChrisClark »

themafia wrote: post #117
ckcheng wrote: post #69

Earlier quoted context omitted.

What’s more funny to me is the set up to that quote: > Bill Gates had somehow manifested, alone, surrounded by ten Apple employees. … Steve started yelling at Bill, asking him why he violated their agreement. And what’s more interesting is the conclusion: > Apple filed a monumental copyright lawsuit against Microsoft in 1988, but they eventually lost on a technicality (the judge ruled that Apple inadvertently gave Mi…

Two spoiled rich kids arguing over who's morality is the least worst. That this moment is held up as some great exchange in business is annoying. That our regulatory agencies are perennially sleep at the switch and allow this nonsense to keep happening is extremely frustrating.

Held up as some great exchange? No it's two assholes arguing with each other. Just like most Jobs documentaries show him as a terrible person.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#130

Post by moritzwarhier »

hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

If this is true, it really would be impressive.
Post reply on HN