This is a concerning pattern. Rebranding merged models as "homegrown" without disclosure undermines trust in open-source AI development. The community needs better provenance tracking and transparency standards for model releases.
Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
201–210 of 261 posts
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#202Can someone please explain or link to some information about how models are merged? Is this genuinely merging weights mathematically or some kind of distillation (presumably not if they’ve done zero training as the post suggests).
There’s nothing to read. Model A: A_1, …, A_n Model B: B_1, …, B_n C_i = A_i * p + B_i * (1 - p) In other words, it’s just a linear combination of the other models’ weights, per position.
It's been a while since I looked at neural networks in detail. Do all the large models have a close enough architecture that this makes sense? Do they have the same number of layers and width? I had thought that each model it's own "secret sauce" of normal and special layers (convolution, max-pooling, something-something) stacked together. Genuinely curious.
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#203> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.
This is called linear mode connectivity and seems to work for almost every large model. So well that in most cases it’s an explicit part of the training process; do many training ‘branches’ then merge then continue. It is not understood why it works so well.
is that actually how they train them in the datacenter? the trillion sized weight vector gets cloned and sent off to groups of GPUs and averaged after?
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#204The municipality of Rio de Janeiro (via its IT company IplanRIO) released Rio-3.5-Open-397B, presented as a homegrown Qwen3.5 fine-tune that beats comparable open models on benchmarks. The linked issue argues it's actually a weighted merge of ~60% Nex-N2 Pro + ~40% Qwen3.5-397B-A17B - Nex-N2 having been released about a week earlier.
Rio better have the best IT infrastructure and software in the world if they are spending time on LLMs. What a waste of tax payer money.
Piaui state it's also doing a LLM it seems. But indeed it would make more sense if it was a national thing rather than local...
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#205Earlier quoted context omitted.
Regarding #2 https://news.ycombinator.com/item?id=48529544
This should be at the top: they uploaded the wrong model, they fixed it
They did upload the wrong model but as of the time of writing they have not fixed it. Right now, 12 hours after they took the old one down, there is simply no model present in their huggingface repo.
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#206Earlier quoted context omitted.
Worth reminding everyone that Lua was also created in Rio, though admittedly at PUC rather than by the government. Rio has a strong engineering talent pool, along with many other major capitals in Brazil
Brazil does have talent. Mauro Carvalho Chehab is a Linux kernel maintainer. Elixir was created by José Valim, a brazilian. I have also created my own programming language. What Brazil doesn't have is a history of properly rewarding talent, which often causes it to migrate elsewhere. So it's definitely surprising when any sort of technological development happens in Brazil: it implies someone who stayed managed to ge…
> extremely high taxes
I always find this funny. Brazilian taxes are nowhere near what I would say “high”. I pay about twice as much out of my compensation as I would pay in Brazil, and that would be as if I did zero tax optimisation back then.
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#207Earlier quoted context omitted.
This should be at the top: they uploaded the wrong model, they fixed it
They did upload the wrong model but as of the time of writing they have not fixed it. Right now, 12 hours after they took the old one down, there is simply no model present in their huggingface repo.
I guess they will upload it later, it seems like an honest mistake to me.
Anyways SwiTransformer paper looks interesting and doing a post training to optimize for it looks interesting as well.
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#208Earlier quoted context omitted.
Brazil does have talent. Mauro Carvalho Chehab is a Linux kernel maintainer. Elixir was created by José Valim, a brazilian. I have also created my own programming language. What Brazil doesn't have is a history of properly rewarding talent, which often causes it to migrate elsewhere. So it's definitely surprising when any sort of technological development happens in Brazil: it implies someone who stayed managed to ge…
> extremely high taxes I always find this funny. Brazilian taxes are nowhere near what I would say “high”. I pay about twice as much out of my compensation as I would pay in Brazil, and that would be as if I did zero tax optimisation back then.
I can second this.
Compared to many countries Brazil doesn't have such high taxes (I'd say that if you work remotely for a company outside of Brazil, you'll probably have much lower taxes compared to almost any other country -- working locally the difference isn't as big, but you have higher taxes in many other places).
What it really lacks is access to capital (which is the real "mojo" of the US compared to the rest of the world).
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#209Earlier quoted context omitted.
Abliterarion is a brute force technique that removes or silences parts of the model. It reduces performance because the abliterated elements aren’t perfectly isolated to censorship so other aspects suffer. Many of the “uncensored” model providers also do some fine tuning on the models. Some of them target better benchmarks or other measures, but outside of the benchmarks and metrics they’re fine tuned for they are ge…
The kind of abliteration you are mentioning is no longer state of the art or the most common form of removing the refusal layer in most models. Your your understanding was up to date about a year and a half ago, but has been out of date since after that.
What OP is describing wasn't called abliteration at all.
Abliteration whilst a neologism implies a surgical ablation of refusal.
Earlier approaches post–trained the model to refuse less and, much like other kinds of fine–tuning, it degraded performance. They were "uncensored".
Abliteration has seen some improvement to this day but it always was close to equivalent performance to the original when compared to those earlier techniques.
Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model
#210Earlier quoted context omitted.
Brazil does have talent. Mauro Carvalho Chehab is a Linux kernel maintainer. Elixir was created by José Valim, a brazilian. I have also created my own programming language. What Brazil doesn't have is a history of properly rewarding talent, which often causes it to migrate elsewhere. So it's definitely surprising when any sort of technological development happens in Brazil: it implies someone who stayed managed to ge…
> extremely high taxes I always find this funny. Brazilian taxes are nowhere near what I would say “high”. I pay about twice as much out of my compensation as I would pay in Brazil, and that would be as if I did zero tax optimisation back then.
As an employee: your taxes are not that high, but public services are terrible so most of middle-class ends up paying for the private alternative as well.
As a business owner: not so bad if you are a freelancing or just a few business partners providing some type of service, but terrible the moment you start considering employing other people.