Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Norway's 2 petabytes of Huawei flash storage and LLM training

blocksandfiles.com

Post reply on HN
Print view View original

171–180 of 227 posts

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#171

Post by gizajob »

WatchDog wrote: post #57
KeplerBoy wrote: post #54

How true is this statement: "He asserted that any country with its own language that did not have a sovereign LLM trained in that language was at a disadvantage as a globally trained, English-speaking LLM would not know about that country’s history, news and culture that was described in the local language." I thought all big players already train on basically everything remotely available to them no matter the langu…

If you want LLMs to have knowledge of the Norwegian language, wouldn't the most obvious thing to do be to build a good training dataset and make the dataset widely available? Why go to the expense of training your own model, especially when it will be inferior to state of the art models.

Because you have so much money you don’t know what to do with it any more.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#172

Post by social_quotient »

KaiserPro wrote: post #153
black_puppydog wrote: post #72

Earlier quoted context omitted.

I task GPT/Claude with researching stuff that pertains to very specific cultural or legal aspects in French politics, on a daily basis. Even though French is a way more common language globally than Norwegian, these models still haven't figured out that, no matter the language I myself speak to them (German or English depending on my mood) their web searches need to be done in French to return reasonable results. I h…

I've noticed that it also imposes american moral judgements on certain things, even though it reasons (sometimes) in the native language. I was trying to work out how and when to use swear words, and the relative power index of them. it translated english swear words into the target language then lectured me on not using them. It took a bunch of prodding for it to actually think as the target language to then get the…

Would be curious about the model and the prompt for this.

Not kidding at all. I had a similar issue with a project where I needed to classify images into specific demographics, and Gemini, while capable, was entirely not going to do the task… until in my JSON response I left room for it to tell me why this was not a good idea and why it was culturally insensitive. Then boom… full JSON array: hair color, eye color, skin color, fitness level, likely ethnicity, likely country of origin, and about 10 other values.

You’re probably wondering what on earth I was working on. I was matching Ai gen headshots to Ai voices so that in an app the voice picker had human (Ai) faces.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#173

Post by varjag »

joe_mamba wrote: post #160
NorwegianDude wrote: post #158

Earlier quoted context omitted.

Considering the fact that the US is complaining about Norway putting too much money into the US market, imagine what would happen if all that money was spent in Norway. It would be chaos.

> imagine what would happen if all that money was spent in Norway. It would create jobs, sovereignty, intellectual property and soft power? Instead it goes to strengthening the tech monopoly of a country that threatens to invade your neighbour.

It was tried in early 1980s and nearly drove any non oil-related industry in the country extinct.

Norway has a manpower bottleneck. The UK had spent its oil windfall domestically and it barely registered. But for a nation of then some 4 million the economy melts down with so much monetary mass.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#174

Post by konschubert »

calgoo wrote: post #168
timmg wrote: post #32

I wonder if instead (or in parallel), Norway should build a set of training data and share it (for free) with all the model builders. Seems like making the frontier models know Norwegian and their culture is a better (or additional!) way to reach the end they are going for here.

Why should they share all this data with the greedy american corporations that are stealing everyones data for their own profit? Much better to keep the legal agreement with the national institutions and possibly develop something actual useful to their own country.

You are contradicting yourself. If you're hoarding the data for yourself you're not going to develop something useful. Sharing the data means that it will be integrated into the big LLMs, which will be useful "for their own country".

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#175

Post by varjag »

amarant wrote: post #74
KeplerBoy wrote: post #54

How true is this statement: "He asserted that any country with its own language that did not have a sovereign LLM trained in that language was at a disadvantage as a globally trained, English-speaking LLM would not know about that country’s history, news and culture that was described in the local language." I thought all big players already train on basically everything remotely available to them no matter the langu…

Not remotely true in my estimation. I don't really speak Norwegian, but I do speak Swedish(which means I mostly understand Norwegian as they're very similar). Every model I've tried speaking Swedish to does it perfectly. I'd be surprised if the same isn't true for Norwegian already

Not really. For instance Facebook speech recognition models had Swedish support but no Norwegian.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#176

Post by NonHyloMorph »

dakolli wrote: post #64

Even entire governments are captured by a mild LLM psychosis. Which is sad in the case of Norway. I lived in Norway for two years and always found their government to be highly rational, this is not a rational use of public funds (but I suppose they have plenty of capital). Western society is completely captured by this form of psychosis and its going to bite us in the a* very soon. I firmly believe all the Boomer le…

I think it is highly rational. You see it from the wrong point of view. It seems to be less a short utilitarian project or economic endeavour, but a cultural one. Think about it more of in terms of applied humanities. Which languages go extinct, which cultures disappear and are superseded by a monocultural globalist hegemony.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#177

Post by embedding-shape »

makeitdouble wrote: post #92
embedding-shape wrote: post #65

Earlier quoted context omitted.

Yeah, was about to comment that too, instead of training a new model and new weights exclusively for Norwegian (and expecting/wanting every other small/medium-sized country to do the same) which seems infinity harder, they could have made high quality transcriptions and translations of the stories currently described only in Norwegian into English, and making it all public. I guess there still would be a worry that i…

> high quality transcriptions and translations of the stories currently described only in Norwegian into English You make it sound like an easier task than training an LLM. I'd argue it's not obvious, and would assume the contrary.

Yes, why wouldn't it be easier to transcribe and translate, skills humanity had for centuries, compared to LLMs that we've only learnt to build these last few years, and even require a frikken computer to do? Of course one of these is harder than the other...

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#178

Post by varjag »

Barrin92 wrote: post #79
orbital-decay wrote: post #71

Earlier quoted context omitted.

Current-best models are pretty fluent at major languages and cultures, so it's untrue at least for the "any" qualifier. Performance is barely affected or might be even better sometimes. However English patterns can subtly leak into native patterns of other languages. It's obviously very different for low-resource languages, but to improve them you need more data, not a new model.

>Current-best models are pretty fluent at major languages and cultures strong disagree on that one. As a German interacting with ChatGPT, even in German it gives me the feeling of talking to the Pluribus people, which reminds me of an anecdote of Walmart failing in Germany because people were freaked out by the constantly upbeat, smiling employees. Understanding a culture is a very different task than translating the…

Set the personality to 'Robot', it makes the interactions so much more tolerable.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#179

Post by RobotToaster »

black_puppydog wrote: post #72
WatchDog wrote: post #57

Earlier quoted context omitted.

If you want LLMs to have knowledge of the Norwegian language, wouldn't the most obvious thing to do be to build a good training dataset and make the dataset widely available? Why go to the expense of training your own model, especially when it will be inferior to state of the art models.

I task GPT/Claude with researching stuff that pertains to very specific cultural or legal aspects in French politics, on a daily basis. Even though French is a way more common language globally than Norwegian, these models still haven't figured out that, no matter the language I myself speak to them (German or English depending on my mood) their web searches need to be done in French to return reasonable results. I h…

Have you tried asking it to translate the prompt to French, and then feeding it the translated prompt?

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#180

Post by onion2k »

WatchDog wrote: post #57
KeplerBoy wrote: post #54

How true is this statement: "He asserted that any country with its own language that did not have a sovereign LLM trained in that language was at a disadvantage as a globally trained, English-speaking LLM would not know about that country’s history, news and culture that was described in the local language." I thought all big players already train on basically everything remotely available to them no matter the langu…

If you want LLMs to have knowledge of the Norwegian language, wouldn't the most obvious thing to do be to build a good training dataset and make the dataset widely available? Why go to the expense of training your own model, especially when it will be inferior to state of the art models.

wouldn't the most obvious thing to do be to build a good training dataset and make the dataset widely available?

Only if you believe other people will value that enough to expend the effort necessary to use it. If you believe other people will see it as low value and ignore it then you'd be better off doing the training yourself in order to guarantee it happens.

There's also a secondary benefit that your team doing the work will learn some useful skills while they do it.

Post reply on HN