Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

github.com

Post reply on HN
Print view View original

171–180 of 261 posts

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#171

Post by s1artibartfast »

rafaquintanilha wrote: post #121

I have no affiliation with them but here's what I think happened: 1. They claim the official model is based on Qwen 397B. It's likely they didn't disclose Nex Pro at all because Nex itself is based on the same base model (not saying they shouldn't). 2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all. 3. It's…

My understanding is that they didnt do any distalation. Tevery weight is a 60/40 element wise average of QWEN and NEX. Is this possible if the rio contracter did thei own post-training as claimed?

https://x.com/tenobrus/status/2066243352211996728/photo/1

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#172

Post by s1artibartfast »

zinodaur wrote: post #10

Oh no, someone is profiting off of their work without proper attribution!?!?

How do you feel about the government or government contractors saying they did a bunch of work when they did nothing instead?

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#173

Post by ls612 »

yowlingcat wrote: post #168
Aurornis wrote: post #153

Earlier quoted context omitted.

Abliterarion is a brute force technique that removes or silences parts of the model. It reduces performance because the abliterated elements aren’t perfectly isolated to censorship so other aspects suffer. Many of the “uncensored” model providers also do some fine tuning on the models. Some of them target better benchmarks or other measures, but outside of the benchmarks and metrics they’re fine tuned for they are ge…

The kind of abliteration you are mentioning is no longer state of the art or the most common form of removing the refusal layer in most models. Your your understanding was up to date about a year and a half ago, but has been out of date since after that.

Nowadays it is that Heretic tool is it not? I’ve seen Gemma models uncensored with it.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#174

Post by alxndresp »

smus wrote: post #169
rafaquintanilha wrote: post #121

I have no affiliation with them but here's what I think happened: 1. They claim the official model is based on Qwen 397B. It's likely they didn't disclose Nex Pro at all because Nex itself is based on the same base model (not saying they shouldn't). 2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all. 3. It's…

What do you mean World Cup debut? haven't they won 5?

They meant their first, opening game of this current World Cup tournament

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#175

Post by dghlsakjg »

_3u10 wrote: post #8
elzbardico wrote: post #3

[flagged]

No, typically Brazilians go to Paraguay for their education, most of their technology comes from Paraguay too.

There’s more than 6x more Brazilian degree holders than there are Paraguayans in total.

That’s a pretty impressive accomplishment.

If true.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#176

Post by dghlsakjg »

bachmeier wrote: post #94
dghlsakjg wrote: post #31

Earlier quoted context omitted.

That’s the joke.

It isn't. The entirety of the comment I responded to is "Oh no, someone is profiting off of their work without proper attribution!?!?" It's a valid point, but references someone using content created by others for profit. I'm objecting to equating this project with the work done by the original content creators. They're not remotely the same thing. I understand how the internet works and how people respond to others…

> It isn’t

It is.

> I understand how the internet works and how people respond to others in this type of setting, but the comment I replied to did not in any way make the point I was making about the disproportionate nature of relative contributions.

Do you understand?

Jokes aren’t that funny when you have to dig into an explanation on the nuance of why the hidden meaning doesn’t match the surface meaning in exact degree and proportions. That turns a joke into a pedantic comment. And paradoxically muddies the point by explaining it.

We aren’t morons. We understand that Picasso is doing something on a different level than someone feeding bulk scraped JPGs of paintings into a python script. You really don’t have to explain.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#177

Post by brunoarueira »

dormento wrote: post #147
diego_moita wrote: post #82

Earlier quoted context omitted.

It is a recurrent Brazilian meme: Rio is known in Brazil as "terra de bandido" (gangster's land). The majority of their politicians have ties to organized crime. There is a virtual revolving door between police and crime, where people migrate from one to the other. It is like Chicago in the 20s, Naples and Medelin in the 80s or Moscow and Culiacan (Sinaloa, Mexico) today.

Rio is kinda funny as a litmus test - federal government creates laws to try and curb some of the corruption, and Rio produces better and better corrupts - so far Rio is winning. BTW wasn't it a few months ago the current governor wanted to leave to be able to run as a candidate, so he asked a supreme justice to step in in as governor, since there wasn't anyone else that technically could?

No, he left to be a Senate candidate and their vice governor left in 2025 to another role, then the next in line is the Legislative Assembly of the State of Rio de Janeiro president, but him was jailed and away from the role. So the next is a judge from the Justice Tribunal.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#178

Post by tredre3 »

monster_truck wrote: post #144
Aurornis wrote: post #97

Earlier quoted context omitted.

> A simple linear combination of every weight did not degrade the performance of the model, but enhanced it. Enhanced it on a couple benchmarks, supposedly. The game is to turn knobs until you get a benchmark run that shows an improvement, then ship it. There are a lot of fine tunes and chimera models on HuggingFace that are supposedly better at some specific test, but when you use them for anything else they're usua…

I don't think your last point is correct. Ablation, when done correctly, seems to increase the quality and typically also the performance too.

That is something often claimed by heretics. My experience couldn't diverge more, however. All heretic (and abliterix) models I've tried are worse than the original. It's not immediately obvious if all you do is ask 2-3 questions and marvel at how it didn't refuse, but try using them for real over longer 8k+ contexts and it falls apart real fast.

They're more prone to getting stuck in loops, becoming unresponsive, and hallucinating more (presumably because of the reduced desire to not answer).

I've tried all the popular heretic peddlers, but if you have one that you can vouch for maybe I've simply missed it.

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#179

Post by antonvs »

meindnoch wrote: post #100
hintymad wrote: post #66

> Every weight tensor in Rio is, to thousands of standard deviations, the same 0.6/0.4 blend of Nex and Qwen — across all 60 layers and every component of the network. Other finetunes cannot be explained as interpolations. I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.

It shows that LLMs are an extremely wasteful approach to intelligence.

Compared to what?

Re: Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model

#180

Post by mathattack »

airstrike wrote: post #160
matheusmoreira wrote: post #133

Earlier quoted context omitted.

I'm honestly impressed that this even happened at all. "Rio de Janeiro's homegrown LLM" is probably the last headline I ever expected to read on HN.

Worth reminding everyone that Lua was also created in Rio, though admittedly at PUC rather than by the government. Rio has a strong engineering talent pool, along with many other major capitals in Brazil

Yes. Though even more than the US, their engineering talent from top schools heads into consulting and finance.
Post reply on HN