Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Norway's 2 petabytes of Huawei flash storage and LLM training

blocksandfiles.com

Post reply on HN
Print view View original

161–170 of 227 posts

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#161

Post by Schlagbohrer »

schubidubiduba wrote: post #134
cortesoft wrote: post #89

Earlier quoted context omitted.

Wouldn't those people be asking the questions in their own language in the first place? The model will reply in the language you use. This thread is about people asking for information about a language that is not the one they are messaging the LLM in

Even if the model will reply in my language, I often notice it searching in english. Or thinking in english. There's always something lost in translation. Sometimes it's just minor nuances. Other times it mangles the legal facts with those of other countries.

This sounds like the problem of people calling "911" as the emergency number which they see in so much US-American media but which is not the emergency number in their own country.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#163

Post by Schlagbohrer »

speedgoose wrote: post #35
solenoid0937 wrote: post #11

> The Olivia system is an HPE Cray Supercomputing EX system, with 448 GPUs and 64,512 CPU cores. Training a sovereign LLM with this meager hardware as opposed to a LORA on some open source model seems like a huge mistake and a potential red flag. There is no way these people have the resources to train a fully fledged LLM, so claiming that is their goal makes me think they don't intend for the LLM to be useful. Which…

They successfully have made PoC finetunes before, so the next step is training fully fledged LLMs. I don’t think they aim to anything worthwhile. The finetunes were incredibly broken. I’m guessing it’s more about having the method to do it. I’m not convinced it’s super useful but I’m not one to decide who gets to do what with the research funds. One finetune I tried did make fun of humans expressing their feelings in…

The article's slides mention how much of an engineering challenge it is just for them to clean their data and create new hardware and software flows to use the data for training. So perhaps it is a big learning exercise to build up institutional / national knowledge of LLM creation.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#164

Post by Schlagbohrer »

KaiserPro wrote: post #156
solenoid0937 wrote: post #11

> The Olivia system is an HPE Cray Supercomputing EX system, with 448 GPUs and 64,512 CPU cores. Training a sovereign LLM with this meager hardware as opposed to a LORA on some open source model seems like a huge mistake and a potential red flag. There is no way these people have the resources to train a fully fledged LLM, so claiming that is their goal makes me think they don't intend for the LLM to be useful. Which…

> There is no way these people have the resources to train a fully fledged LLM, so claiming that is their goal makes me think they don't intend for the LLM to be useful. Depends on what they are doing and why. but at most big labs, only the final model training happens on the big clusters. a lot of experimentation happens on So for fast iteration, this seems fine.

This is the use case for the small NVIDIA boxes that a researcher can have on their desk for $5k and do useful experiments before spending all the grant money on a huge training run for the final product.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#165

Post by TrackerFF »

vidarh wrote: post #52
TrackerFF wrote: post #14

I'm a Norwegian, and I use the national library almost every day for searching through texts. They have truly one of the best working user interfaces (and functionality) for searching through the massive amounts of text.

It's really fantastic. I just wished there were fewer restrictions on the content that is accessible. (a lot is only accessible from Norwegian IP addresses, so it's one of the main reasons I maintain a VPN as I'm Norwegian but live in the UK; a second set is only available from the IP addresses of libraries or research institutions - still huge amounts that are generally available, though)

My biggest gripe with it are the restrictions, indeed.

When searching through the closed newspapers, you have to apply for access manually, which gives you 8 hours of access. Great. Only that the access is seemingly manually granted - so if you apply 16:05 on a Friday, chances are you won't get any access until 9-10 the next Monday.

With that said, I do understand why it is like that. If people could apply via API, and get instant access, they would probably just stop buying newspaper subscriptions.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#166

Post by bakugo »

black_puppydog wrote: post #72
WatchDog wrote: post #57

Earlier quoted context omitted.

If you want LLMs to have knowledge of the Norwegian language, wouldn't the most obvious thing to do be to build a good training dataset and make the dataset widely available? Why go to the expense of training your own model, especially when it will be inferior to state of the art models.

I task GPT/Claude with researching stuff that pertains to very specific cultural or legal aspects in French politics, on a daily basis. Even though French is a way more common language globally than Norwegian, these models still haven't figured out that, no matter the language I myself speak to them (German or English depending on my mood) their web searches need to be done in French to return reasonable results. I h…

> their web searches need to be done in French to return reasonable results.

I wonder how much of this is also just the search engine's region setting.

It's a big problem I regularly have with Google. I almost always want English language, US-centric results, so I have my region set to the US. But occasionally I want results relevant to my actual country, and even searching in my native language usually yields much worse results than just opening an incognito tab and letting it default to my real location.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#168

Post by calgoo »

timmg wrote: post #32

I wonder if instead (or in parallel), Norway should build a set of training data and share it (for free) with all the model builders. Seems like making the frontier models know Norwegian and their culture is a better (or additional!) way to reach the end they are going for here.

Why should they share all this data with the greedy american corporations that are stealing everyones data for their own profit? Much better to keep the legal agreement with the national institutions and possibly develop something actual useful to their own country.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#169

Post by skissane »

Schlagbohrer wrote: post #161
schubidubiduba wrote: post #134

Earlier quoted context omitted.

Even if the model will reply in my language, I often notice it searching in english. Or thinking in english. There's always something lost in translation. Sometimes it's just minor nuances. Other times it mangles the legal facts with those of other countries.

This sounds like the problem of people calling "911" as the emergency number which they see in so much US-American media but which is not the emergency number in their own country.

I remember being bored as a teenager on a family holiday to New Zealand in the 1990s, so I went and dialled 911 from a payphone to see what would happen-I got a recorded message saying that in New Zealand, the emergency number isn’t 911, it is 111. Dialling 000 (the Australian emergency number) produced a similar recorded message.

Re: Norway's 2 petabytes of Huawei flash storage and LLM training

#170

Post by amelius »

KeplerBoy wrote: post #54

How true is this statement: "He asserted that any country with its own language that did not have a sovereign LLM trained in that language was at a disadvantage as a globally trained, English-speaking LLM would not know about that country’s history, news and culture that was described in the local language." I thought all big players already train on basically everything remotely available to them no matter the langu…

It's probably just an excuse to play with LLMs using big government funding :)
Post reply on HN