Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: I want to train a LM on my home country's dialect, how can I do it?

news.ycombinator.com

Post reply on HN
Print view View original

11–13 of 13 posts

Re: Ask HN: I want to train a LM on my home country's dialect, how can I do it?

#11

Post by the_generalist »

ktrnka wrote: post #10
ktrnka wrote: post #9

I'd suggest starting with just building a high quality data set with text from a variety of domains, and starting off by publishing that. Maybe even developing some related tech like adding the dialect to language id packages. Another key thing might be to build a nicely curated word list for the dialect, and make sure there's good documentation for researchers wanting to work in the language. Partly I'm feeling insp…

Also I'm happy to help any way I can. I'm not sure the best practices of sharing contact info on HN but if you Google K Trnka language modeling I should be the only one

That's great, I think I've added you on LinkedIn

Re: Ask HN: I want to train a LM on my home country's dialect, how can I do it?

#12

Post by barrenko »

yorwba wrote: post #4

If all you want is a LM and it doesn't need to be trained by you or run on infrastructure you control, you could try to see whether ChatGPT already understands well enough. A Tunisian friend of mine told me that he asked it to tell a joke in Tunisian Arabic and it worked, only the joke wasn't funny. If you want or need to train on your own data, social media is a good bet for colloquial language. You could try export…

Do you know of further link or resources in this "direction", this is awesome.
Post reply on HN