Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
1–10 of 55 posts
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#2Is there something similar for romance or Germanic languages?
And how did they decide that, e.g., German or Dutch would make the model worse?
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#3I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#4Is there something similar for romance or Germanic languages? And how did they decide that, e.g., German or Dutch would make the model worse?
I don't think they decided that, they included Finish which is completely unrelated to the other nordic languages. If they just picked languages that are related for cross learning including Dutch or German would have made more sense indeed.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#5I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.
Just like adding code to textual models helps the model develop its reasoning capabilities, it seems like adding more languages helps in other areas too. What is needed is more good quality data to train on...
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#6I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.
Perform better how? Knowing more languages gives you more data and different points of view rather than just using the English corpus and culture. When I ask chatgpt for a translation it seems to understand the meaning behind the words and finds the closest thing in the other language. The datasets seem to merge in some way
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#7Great talking points. These are highly relevant subjects and I'm delighted we in the Nordics are keeping up with current developments. This work is important for preserving our culture.
I hope to see this used to generate a customized curriculum for each neurodiverse child so that we can live in a more equitable society.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#8Is there something similar for romance or Germanic languages? And how did they decide that, e.g., German or Dutch would make the model worse?
I don't think they decided that, they included Finish which is completely unrelated to the other nordic languages. If they just picked languages that are related for cross learning including Dutch or German would have made more sense indeed.
The root is unrelated but Finnish has certainly been shaped by Swedish and Russian and most recently English languages in the last 200 years
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#9I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.
Just like adding code to textual models helps the model develop its reasoning capabilities, it seems like adding more languages helps in other areas too. What is needed is more good quality data to train on...
We also see humans get worse at specific things when they learn too much in general. There is a cut-off point to how many concepts we can learn with what skill. To be most effective, we have to specialize in the right things while continuing to acquire generalist knowledge. It’s a balancing act.
These architectures are less capable than brains in many ways. So, we should expect them to have such trade-offs. An efficient one should work fine on English, mathematical notation, and a programming language. Maybe samples of others that illustrate unique concepts. I’m also curious how many languages or concepts you can add to a given architecture before its effectiveness starts dropping.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#10Would love to know more about their experience training on AMD GPUs. Was it just as seamless as using Cuda?