Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

github.com

Post reply on HN
Print view View original

91–100 of 261 posts

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#91

Post by kolanos »

woadwarrior01 wrote: post #81
hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

It's is a well known idea[1], although it's still surprising that something as simple, even works. [1]: https://arxiv.org/abs/2203.05482

This team could have stopped here and still had something interesting (albeit not novel) to show. But the hype cycle was too tempting.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#92

Post by axus »

fkozlowski wrote: post #18

I'm honestly surprised that they even had the inclination to attempt creating a model. I guess it's bullish that a municipal IT department had the guts to try this?

I like the [dead] comment theory that they proposed a huge LLM training budget to the government, kept most of the money, and released a cheap merge to justify the grift.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#93

Post by randall »

hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

[dead]

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#94

Post by bachmeier »

dghlsakjg wrote: post #31
bachmeier wrote: post #24

Earlier quoted context omitted.

"Their work"? First you had the original content creators that did 99.99% of the work. Then you had the US companies bundle it up into a frontier LLM. Then "they" did the "work" of using the US model as a foundation for their own. So in the sense of doing 0.00001% of the actual work that went into their product, sure. I'd say it's more like someone forking a Linux distro, adding a few themes and fonts, and then compl…

That’s the joke.

It isn't. The entirety of the comment I responded to is "Oh no, someone is profiting off of their work without proper attribution!?!?" It's a valid point, but references someone using content created by others for profit. I'm objecting to equating this project with the work done by the original content creators. They're not remotely the same thing.

I understand how the internet works and how people respond to others in this type of setting, but the comment I replied to did not in any way make the point I was making about the disproportionate nature of relative contributions.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#95

Post by serial_dev »

Aurornis wrote: post #55
Lucasoato wrote: post #40

Earlier quoted context omitted.

So the problem isn’t in the missing attribution to Qwen, but with the fact that they didn’t mention Nex-N2 Pro right?

The problem is that they claimed to have made a big achievement with their home grown post training, and they expected to receive a lot of praise for it. Then researchers looked at the weights and there is no post training at all. They are now attributing both models they merged, but their excuse for the lack of post training is to claim they accidentally uploaded the wrong files.

I’d believe they accidentally uploaded the wrong files if they uploaded the correct ones. To state that they accidentally uploaded something else and then not upload the correct version means they probably do not have anything and either hope people forget about this or they are scrambling to have something that is at least close to their original claim.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#96

Post by x312 »

hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

This works because Nex itself is a finetune of Qwen3.5 (https://huggingface.co/nex-agi/Nex-N2-Pro). It's merging Qwen3.5 with a Qwen3.5 finetune.

I don't believe this would work on two LLMs that have different pretraining. Even if it did you would need two LLMs that have exact same internal activation shapes, dimensions, expert counts, token vocabulary, realistically it would never happen outside of finetunes or academic experiments.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#97

Post by Aurornis »

hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

> A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

Enhanced it on a couple benchmarks, supposedly.

The game is to turn knobs until you get a benchmark run that shows an improvement, then ship it. There are a lot of fine tunes and chimera models on HuggingFace that are supposedly better at some specific test, but when you use them for anything else they're usually worse.

This happens with a lot of the models that are modified to remove censorship. They succeed in getting the model to emit previously censored outputs, but the overall output quality decreases.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#98

Post by yieldcrv »

daemonologist wrote: post #44
yieldcrv wrote: post #21

Didn’t the last thread about this have someone from the lab or an enthusiast in Rio saying exactly that? Its a fine tune of Qwen Not a conspiracy

The allegation here is that it's not actually a fine-tune of Qwen, but instead an undisclosed mashup (merge) of someone else's fine-tune of Qwen and the original model. Rio subsequently said that the model was in fact a merge, that they did additional fine-tuning after the merge, and that they accidentally uploaded the base merge instead of the version with additional fine-tuning. But this seems like quite an oversig…

> But this seems like quite an oversight...

Not to me, what would people like to happen? Who are those people? And why do they care?

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#99

Post by vips7L »

internet2000 wrote: post #11
zinodaur wrote: post #10

Oh no, someone is profiting off of their work without proper attribution!?!?

Attribution isn't the relevant part. Lying about your lab's capabilities is.

Sounds like the whole AI movement.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#100

Post by meindnoch »

hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

It shows that LLMs are an extremely wasteful approach to intelligence.
Post reply on HN