5 million for a llama-2 finetune, how is that impressive?
AMÁLIA and the future of European Portuguese LLMs
41–50 of 90 posts
Re: AMÁLIA and the future of European Portuguese LLMs
#42This model is a waste of Public Funds. There is no public website to use it, be it free or paid, the dataset is not public, the code is not public (The github URL in the article returns 404 ), the claimed model intelligence is so low that is pretty much useless at 32K context and massively inferior to GPT‑4o. As per tradition in Portugal, some people managed to get 5.5 Million to produce nothing and no one is asking…
It’s a way to suck all the money out of the room in the name of nationalism — and it’s all over Europe. Only idea everyone has had.
Re: AMÁLIA and the future of European Portuguese LLMs
#43It is definitely an interesting problem, because Portugal is a small enough country that the actual total corpus of available texts in (non-Brazilian) Portuguese is potentially problematic.
I don't think so, Portugal the country might be small, with a small population, but there is ~250 million "Lusophones" (native Portuguese speakers), making it the fifth-most spoken native language in the world, I'd hardly call that small :) And before everyone screams; yes, European Portuguese is different from Brazilian Portuguese, but they're still both Portuguese and understand each other, so it's not like the tex…
Portugal has a growing Xenophobic attitude towards immigrants, specially Brazilians and this is reflected in linguistic prejudice.
They have concerns of portuguese children learning to "speak brazillian" because there is a lot more of video content being produced in Brasil than in Portugal and stuff like movies, videogames and software in general are avaliable in brazilian localization/adaptation first.
Re: AMÁLIA and the future of European Portuguese LLMs
#44It is definitely an interesting problem, because Portugal is a small enough country that the actual total corpus of available texts in (non-Brazilian) Portuguese is potentially problematic.
European Portuguese is the 13th most populous language in Europe. Not that small, there are many other European languages in use that are much smaller. https://en.wikipedia.org/wiki/List_of_languages_by_number_of...
What makes Portugal's situation unique is that it is a small population that is eclipsed in models by the bigger weights of the much bigger population of Brazil.
Yes, there are much smaller European countries, but those are generally the only source of truth for their specific language, so the context of a LLM query in that language steers the LLM towards facts from that country, for example, if I ask a big generic LLM something in Latvian then it most likely will answer something relevant to the context of Latvia. But Portugal, being the much smaller user of its language, have the somewhat unique problem that if I ask a generic model something in Portuguese it will probably answer something related to Brazil instead of Portugal.
Maybe the UK and Spain have somewhat similar struggles, but I suspect that none has it as bad as Portugal in that regard.
Re: AMÁLIA and the future of European Portuguese LLMs
#45"This model is a waste of Public Funds". There is no "public funds", this is a waste of money from the tax payers.
Re: AMÁLIA and the future of European Portuguese LLMs
#46Earlier quoted context omitted.
I don't think so, Portugal the country might be small, with a small population, but there is ~250 million "Lusophones" (native Portuguese speakers), making it the fifth-most spoken native language in the world, I'd hardly call that small :) And before everyone screams; yes, European Portuguese is different from Brazilian Portuguese, but they're still both Portuguese and understand each other, so it's not like the tex…
Portugal has a growing Xenophobic attitude towards immigrants, specially Brazilians and this is reflected in linguistic prejudice. They have concerns of portuguese children learning to "speak brazillian" because there is a lot more of video content being produced in Brasil than in Portugal and stuff like movies, videogames and software in general are avaliable in brazilian localization/adaptation first.
We have the same thing happening, on multiple levels, here too. First some Spanish parents are afraid the children aren't listening and watching enough Spanish media. Then additionally, some Catalan parents are afraid the children don't get to use Catalan in school so they don't become proficient enough to use it in society.
Re: AMÁLIA and the future of European Portuguese LLMs
#47Earlier quoted context omitted.
Evidence? Not so much, I didn't realize I was defending a PhD thesis here. I speak Spanish, and have talked with people who only speak Portuguese, either of the variants, and also talked with Portuguese people before how they see their language, comparing it with Brazilian Portuguese, and vice-versa. So basically based on vibes and experience. > And if the first 80% doesn't bias the language after post-training (whic…
> I'm not sure how many languages you speak or encountered in the wild before, but some languages are VERY different from each other, some are a bit different and others are basically the same with some differences. I'm a dual citizen of Portugal and Brazil and I live in the US now, so that's my linguistic background. (Also studied bits of French, Russian, Latin and Greek.) > Doing what I describe for languages that…
> If the goal is to create an LLM with minimal Brazilian Portuguese bias (which was one of their main goals)
Oh, I wasn't aware that was their goal, would certainly be intuitive to avoid Brazilian Portuguese if that's the case, although I'm still not sure it actually makes sense to 100% avoid it for pre-training even if you're trying to avoid Brazilian bias, you can "skew" things pretty heavily in post-training if you so wish.
Where can I read more about this goal, because it doesn't seem to be mentioned in the submission article, just a short off-hand about one of the benchmarks, so I'm guessing there is some resource they talk more about the specifically perhaps?
Re: AMÁLIA and the future of European Portuguese LLMs
#48Earlier quoted context omitted.
Portugal has a growing Xenophobic attitude towards immigrants, specially Brazilians and this is reflected in linguistic prejudice. They have concerns of portuguese children learning to "speak brazillian" because there is a lot more of video content being produced in Brasil than in Portugal and stuff like movies, videogames and software in general are avaliable in brazilian localization/adaptation first.
We have the same thing happening, on multiple levels, here too. First some Spanish parents are afraid the children aren't listening and watching enough Spanish media. Then additionally, some Catalan parents are afraid the children don't get to use Catalan in school so they don't become proficient enough to use it in society.
The Catalan situation is completely different and unrelated, being a completely different language and not endangered (with or without scary quotes, as you prefer) by an ex-colony that became independent. Actually many Catalans would like to be such ex-colony.
Re: AMÁLIA and the future of European Portuguese LLMs
#49This model is a waste of Public Funds. There is no public website to use it, be it free or paid, the dataset is not public, the code is not public (The github URL in the article returns 404 ), the claimed model intelligence is so low that is pretty much useless at 32K context and massively inferior to GPT‑4o. As per tradition in Portugal, some people managed to get 5.5 Million to produce nothing and no one is asking…
I'm not arguing with the rest of your points, but...
> Just fine tune the open source Kimi 2.6 with an open source Portuguese dataset
I think that tokenizers of all popular models are heavily biased towards English or English and Mandarin.
And I don't think that it is possibple to replace the tokenizer without full retraining.
Re: AMÁLIA and the future of European Portuguese LLMs
#50This model is a waste of Public Funds. There is no public website to use it, be it free or paid, the dataset is not public, the code is not public (The github URL in the article returns 404 ), the claimed model intelligence is so low that is pretty much useless at 32K context and massively inferior to GPT‑4o. As per tradition in Portugal, some people managed to get 5.5 Million to produce nothing and no one is asking…
As a pt-BR speaker from across the pond: https://soberania.ai/
Similar waste.