Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

github.com

Post reply on HN
Print view View original

191–200 of 261 posts

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#191

Post by antonvs »

monster_truck wrote: post #144
Aurornis wrote: post #97

Earlier quoted context omitted.

> A simple linear combination of every weight did not degrade the performance of the model, but enhanced it. Enhanced it on a couple benchmarks, supposedly. The game is to turn knobs until you get a benchmark run that shows an improvement, then ship it. There are a lot of fine tunes and chimera models on HuggingFace that are supposedly better at some specific test, but when you use them for anything else they're usua…

I don't think your last point is correct. Ablation, when done correctly, seems to increase the quality and typically also the performance too.

I'm curious about where you got that idea from. Neither the theory nor the available examples support it. If it did, everyone knowledgeable would be using abliterated models.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#192

Post by vasco »

bachmeier wrote: post #94
dghlsakjg wrote: post #31

Earlier quoted context omitted.

That’s the joke.

It isn't. The entirety of the comment I responded to is "Oh no, someone is profiting off of their work without proper attribution!?!?" It's a valid point, but references someone using content created by others for profit. I'm objecting to equating this project with the work done by the original content creators. They're not remotely the same thing. I understand how the internet works and how people respond to others…

> I understand how the internet works and how people respond to others in this type of setting,

You should frame this as a reminder to be more charitable in your positions because sometimes you can be wrong. This subthread ended being one of the funniest I've read recently.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#193

Post by Davidzheng »

hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

it's interesting that this was even guessed at

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#194

Post by jxmorris12 »

jordz wrote: post #73

Can someone please explain or link to some information about how models are merged? Is this genuinely merging weights mathematically or some kind of distillation (presumably not if they’ve done zero training as the post suggests).

There’s nothing to read.

Model A: A_1, …, A_n Model B: B_1, …, B_n

C_i = A_i * p + B_i * (1 - p)

In other words, it’s just a linear combination of the other models’ weights, per position.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#195

Post by Davidzheng »

Davidzheng wrote: post #193
hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

it's interesting that this was even guessed at

ok I guess they had other clues then if you do any sort of comparison vs Nex & Qwen probably a lot of weird coincidences will show up if somehow the three weights are not linearly independent lol

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#196

Post by ipieter »

croes wrote: post #187
Aurornis wrote: post #156

Earlier quoted context omitted.

> 2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all. They merged the base model with another lab’s fine tuned model. The improvements could have come from getting some of the fine tuned weights from the other model. If they really had a better performing model that they “accidentally” forgot to upload, they…

Seems they did https://news.ycombinator.com/item?id=48529544

I only see an edit to the readme (13h ago) and removal of the weights, so the repo is now empty.

I am willing to give them the benefit of the doubt, but we've seen this before: a model gets released that is supposedly state-of-the-art, yet seems to be a an other repackaged model without any training. Reflection 70B was the most similar example, all they now need is an api that rewrites "Claude" to "Rio".

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#197

Post by antonvs »

FooBarWidget wrote: post #190

Can anyone explain to me what a merge is and why that works? It seems utterly bizarre to me that you can just merge weights. You can't make a working program by just merging machine instruction pages. Aren't weights tightly coupled to a specific architecture?

In this case both sets of weights ultimately came from the same model. The Nex model they used is a fine-time of Qwen, which was the other model they used.

I'm not an expert in this area, but it's not too hard to see how a merge like that could turn out ok.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#198

Post by antonvs »

thimabi wrote: post #46
ekjhgkejhgk wrote: post #15

One funny thing about incompetence is that they don't have the competence to know that their incompetence is straightforward to verify by a competent person.

I wouldn’t describe what happened here as incompetence. As a “carioca”, I am pleasantly surprised to know that the government’s IT department is involved in AI work — even without the budget to create its own models from scratch.

They could do AI work without trying to lie to the entire rest of the world.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#199

Post by antonvs »

yieldcrv wrote: post #98
daemonologist wrote: post #44

Earlier quoted context omitted.

The allegation here is that it's not actually a fine-tune of Qwen, but instead an undisclosed mashup (merge) of someone else's fine-tune of Qwen and the original model. Rio subsequently said that the model was in fact a merge, that they did additional fine-tuning after the merge, and that they accidentally uploaded the base merge instead of the version with additional fine-tuning. But this seems like quite an oversig…

> But this seems like quite an oversight... Not to me, what would people like to happen? Who are those people? And why do they care?

They made a public claim to having produced a useful model, which they published. Turns out they did nothing of the sort.

> why do they care?

Why does anyone ever care about having their time wasted by fraudulent claims?

Post reply on HN