Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

github.com

Post reply on HN
Print view View original

181–190 of 261 posts

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#181

Post by hypercube33 »

DonsDiscountGas wrote: post #56
unrvl22 wrote: post #2

The municipality of Rio de Janeiro (via its IT company IplanRIO) released Rio-3.5-Open-397B, presented as a homegrown Qwen3.5 fine-tune that beats comparable open models on benchmarks. The linked issue argues it's actually a weighted merge of ~60% Nex-N2 Pro + ~40% Qwen3.5-397B-A17B - Nex-N2 having been released about a week earlier.

I didn't know model merging like that was possible. (Obviously possible from a pure software standpoint but I'm surprised it's effective)

Even merging models with themselves as shown here in the post how they got to the top of hugging face with two gpus

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#183

Post by xiphias2 »

throwa356262 wrote: post #124
rafaquintanilha wrote: post #121

I have no affiliation with them but here's what I think happened: 1. They claim the official model is based on Qwen 397B. It's likely they didn't disclose Nex Pro at all because Nex itself is based on the same base model (not saying they shouldn't). 2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all. 3. It's…

Regarding #2 https://news.ycombinator.com/item?id=48529544

This should be at the top: they uploaded the wrong model, they fixed it

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#184

Post by itkovian_ »

hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

This is called linear mode connectivity and seems to work for almost every large model. So well that in most cases it’s an explicit part of the training process; do many training ‘branches’ then merge then continue.

It is not understood why it works so well.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#185

Post by philipallstar »

jrm4 wrote: post #111
philipallstar wrote: post #74

Earlier quoted context omitted.

Companies that generate loads of corporation tax, income tax, and VAT revenue are the exact opposite of wastes of public money.

Yes, when they do so proportional to what they take, especially as compared to individuals and their tax liabilities. You'll have to let me know when that finally happens, because that ain't now.

Sorry, I've no idea how to read your first sentence.

Your second one - that's how everything public is paid for. Private individuals pay tax, either through their corporations paying corporation tax or the tax bill on top of their wage bills, which a) drives up prices of the goods and services they offer, or depresses wages, and b) funds all the public sector employees and orgs that don't pay tax (orgs) or don't pay net tax (employees).

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#186

Post by weitendorf »

yowlingcat wrote: post #168
Aurornis wrote: post #153

Earlier quoted context omitted.

Abliterarion is a brute force technique that removes or silences parts of the model. It reduces performance because the abliterated elements aren’t perfectly isolated to censorship so other aspects suffer. Many of the “uncensored” model providers also do some fine tuning on the models. Some of them target better benchmarks or other measures, but outside of the benchmarks and metrics they’re fine tuned for they are ge…

The kind of abliteration you are mentioning is no longer state of the art or the most common form of removing the refusal layer in most models. Your your understanding was up to date about a year and a half ago, but has been out of date since after that.

Unrelated but I’ve been putting off learning about post-abliteration technique and want to use it for an upcoming open source “retraining” project I have on my backlog. I’m not interested in the refusal layers though, more like deep fine tuning but in a way that might let me prune out or consolidate layers, if that makes sense? Do you have any pointers or links to the current SOTA in this area?

I guess I’m looking for a kind of bulk/sticky dropout (which was in fashion way back when I studied DNN in school).

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#187

Post by croes »

Aurornis wrote: post #156
rafaquintanilha wrote: post #121

I have no affiliation with them but here's what I think happened: 1. They claim the official model is based on Qwen 397B. It's likely they didn't disclose Nex Pro at all because Nex itself is based on the same base model (not saying they shouldn't). 2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all. 3. It's…

> 2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all. They merged the base model with another lab’s fine tuned model. The improvements could have come from getting some of the fine tuned weights from the other model. If they really had a better performing model that they “accidentally” forgot to upload, they…

Seems they did

https://news.ycombinator.com/item?id=48529544

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#188

Post by knuppar »

matheusmoreira wrote: post #132
_3u10 wrote: post #8

Earlier quoted context omitted.

No, typically Brazilians go to Paraguay for their education, most of their technology comes from Paraguay too.

No. We go to Paraguay to buy cheaper electronics.

muamba garai

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#189

Post by vasco »

unrvl22 wrote: post #2

The municipality of Rio de Janeiro (via its IT company IplanRIO) released Rio-3.5-Open-397B, presented as a homegrown Qwen3.5 fine-tune that beats comparable open models on benchmarks. The linked issue argues it's actually a weighted merge of ~60% Nex-N2 Pro + ~40% Qwen3.5-397B-A17B - Nex-N2 having been released about a week earlier.

Rio better have the best IT infrastructure and software in the world if they are spending time on LLMs. What a waste of tax payer money.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#190

Post by FooBarWidget »

Can anyone explain to me what a merge is and why that works? It seems utterly bizarre to me that you can just merge weights. You can't make a working program by just merging machine instruction pages. Aren't weights tightly coupled to a specific architecture?
Post reply on HN