double slash in the shared link probably not ideal (though inconsequential)
https://www.silo.ai/blog/viking-7b-the-first-open-llm-for-th...
Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
11–20 of 55 posts
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#12I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.
Just like adding code to textual models helps the model develop its reasoning capabilities, it seems like adding more languages helps in other areas too. What is needed is more good quality data to train on...
I guess you mean non-textual data then because the amount of text data they are being trained on ought to be enough for agi by now?
Some kind of diminishing returns asymptote from text volume alone must have been hit a long time ago.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#13Would love to know more about their experience training on AMD GPUs. Was it just as seamless as using Cuda?
[deleted]
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#14Would love to know more about their experience training on AMD GPUs. Was it just as seamless as using Cuda?
> To leverage the capabilities of MI250X, ROCm enables the use of GPU matrix cores through its rocBLAS and MIOpen library implementations that, in turn,are leveraged by PyTorch.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#15Earlier quoted context omitted.
Just like adding code to textual models helps the model develop its reasoning capabilities, it seems like adding more languages helps in other areas too. What is needed is more good quality data to train on...
I guess you mean non-textual data then because the amount of text data they are being trained on ought to be enough for agi by now? Some kind of diminishing returns asymptote from text volume alone must have been hit a long time ago.
It's not the amount that is wrong, it's how the model is trained. The model is trained for zero and few shot tasks. It is not surprising that it is performing well when you ask for that.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#16Would love to know more about their experience training on AMD GPUs. Was it just as seamless as using Cuda?
They probably got a lot of hand holding from AMD.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#17Although not nordic not including basque which I guess could also be considered an European low-resource language.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#18Is there something similar for romance or Germanic languages? And how did they decide that, e.g., German or Dutch would make the model worse?
I don't think they decided that, they included Finish which is completely unrelated to the other nordic languages. If they just picked languages that are related for cross learning including Dutch or German would have made more sense indeed.
Including Finnish was probably just a political choice, since Finland and Sweden are very close politically, much closer than Germany or other areas with more similar languages.
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#19I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.
they would perform worse, i promise you
Re: Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
#20I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.
Just like adding code to textual models helps the model develop its reasoning capabilities, it seems like adding more languages helps in other areas too. What is needed is more good quality data to train on...
> its reasoning capabilities
To be clear, LLMs are not capable of reasoning.