Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

silo.ai

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

#41

Post by staticman2 »

ClarityJones wrote: post #32
coffeebeqn wrote: post #6

Earlier quoted context omitted.

Perform better how? Knowing more languages gives you more data and different points of view rather than just using the English corpus and culture. When I ask chatgpt for a translation it seems to understand the meaning behind the words and finds the closest thing in the other language. The datasets seem to merge in some way

Fair, but there may be overhead that doesn't need to exist. Certainly - for the limited compute my brain can accomplish - I could gain a deeper understanding of physics, if I focused on learning physics and didn't also have to simultaneously learn French.

Wouldn't a better metaphor be if a child growing up in a bilingual household would be worse at physics as an adult? My guess would be growing up bilingual would have no impact.

Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

#42

Post by anewhnaccount3 »

smokracek wrote: post #35

First thing I notice is that Finnish is part of a completely different language family from the other Nordic languages and English (Uralic vs. Indo-European). I wonder to what extent this affects the effectiveness of their low-resource training. Finnish is highly agglutinative, adding prefixes and suffixes to modify a root. My (amateur) take is that the tokenization and attention patterns may differ a lot? Would love…

Finnish is not so different dispute having different lineages. Even if we talk about morphology, sometimes it's simply that e.g. prepositions are affixed to the end of a word big whoop. There are many dimensions to language vairation. Finnish has a long history of contact with Scandi languages and a lot of borrowed words and logic. It would be good to have Estonian and possibly Baltic languages too.

ETA: It is differentof course just perhaps not as much as people sometimes try to say. You can definitely ruffle some feathers with this one given the uniqueness of Finnish is pretty central to Finnish nationalism.

Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

#44

Post by jug »

If you're interested in this, don't miss AI Sweden's GPT-SW3 @ 126M to 40B trained on Nordic languages (not Finnish) and English. It's funded by the Swedish government and partners, and freely available with a pretty lively Discord for ongoing AI research focusing on the Nordic languages. I think Viking is called "first" because it includes Finnish, because otherwise, GPT-SW3 was released earlier.

https://huggingface.co/AI-Sweden-Models

Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

#45

Post by richdougherty »

bangaladore wrote: post #3

I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.

I can't track down the citation (either Google or DeepMind I think), but I remember reading research from a year or two ago how adding extra languages (French, German) improved English language performance. There may have also been an investigation about multi modality too, which found that adding vision or audio helped with text as well.

Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

#47

Post by larodi »

The fact it was trained on HPC which covers 20% heat consumption in a city is absolutely wild and on par with how wild it is to have English/Nordic model.

“ Further emphasizing digital sovereignty, Viking is trained on the EuroHPC supercomputer LUMI, utilizing up to 4096 AMD MI-250X GPUs. LUMI is not only Europe’s most powerful supercomputer and the 5th most powerful in the world, but also the 3rd greenest supercomputer among the top 500 supercomputers. LUMI’s energy consumption is covered with power produced 100% with hydroelectricity, and the waste heat of LUMI will account for about 20 percent of the district heating in the surrounding city of Kajaani. ”

Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

#48

Post by lostmsu »

jug wrote: post #44

If you're interested in this, don't miss AI Sweden's GPT-SW3 @ 126M to 40B trained on Nordic languages (not Finnish) and English. It's funded by the Swedish government and partners, and freely available with a pretty lively Discord for ongoing AI research focusing on the Nordic languages. I think Viking is called "first" because it includes Finnish, because otherwise, GPT-SW3 was released earlier. https://huggingface…

Why do they do training from scratch instead of starting off LLAMA 3 or something else?

Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

#49

Post by phlip9 »

bangaladore wrote: post #3

I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.

Interesting thought. Maybe an LLM would build deeper insight with only one training language. On the other hand, the model might overfit with just one language -- maybe multilingual models generalize better?

Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

#50

Post by Tor3 »

anewhnaccount3 wrote: post #42
smokracek wrote: post #35

First thing I notice is that Finnish is part of a completely different language family from the other Nordic languages and English (Uralic vs. Indo-European). I wonder to what extent this affects the effectiveness of their low-resource training. Finnish is highly agglutinative, adding prefixes and suffixes to modify a root. My (amateur) take is that the tokenization and attention patterns may differ a lot? Would love…

Finnish is not so different dispute having different lineages. Even if we talk about morphology, sometimes it's simply that e.g. prepositions are affixed to the end of a word big whoop. There are many dimensions to language vairation. Finnish has a long history of contact with Scandi languages and a lot of borrowed words and logic. It would be good to have Estonian and possibly Baltic languages too. ETA: It is differ…

As someone growing up in relatively close contact with Finnish, I can assure you that there's no real common ground between Scandinavian languages (Swedish, Danish, Norwegian) and Finnish. There are loan words, but they are few and far between and in any case does not make for any mutual understanding. I've been so much to Finland that I would really like to learn to at least understand the language, instead of relying of memorized names of foodstuff and the like. Just have to tackle Japanese first.. (and I consider that one an easier operation)
Post reply on HN