Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I trained a 9M speech model to fix my Mandarin tones

simedw.com

Post reply on HN
Print view View original

51–60 of 166 posts

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#51

Post by et-al »

tifan wrote: post #40
dapangzi wrote: post #13

Longtime lurker, made an account specifically to give feedback here as an intermediate speaker. :) This is a great initiative and I hope to see more come out of this; I am not criticizing, but just want to provide my user experience here so you have data points. In short, my experience lines up with your native speakers. I found that it loses track of the phonemes when speaking quickly, and tones don't seem to line u…

I had the same issue! Perhaps being another dapangzi is the problem here lol

I'm not familiar with this slang: what's a big plate?

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#52

Post by hawflakes »

affogarty wrote: post #11

This is extremely cool, although I asked my wife (who is Chinese) to try it out and it said she made some mistakes.

I tried it out and it has some issues with my native speech. I grew up with more Taiwan mandarin but I know the Beijing standard and the recognizer was flagging some of my utterances incorrectly.

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#53

Post by zdc1 »

bunderbunder wrote: post #45

This is very cool, but from one Mandarin learner to another I’d caution against relying too heavily on any external feedback mechanism for improving your pronunciation. If you can’t easily hear your pronunciation mistakes so clearly it hurts, consider putting more energy into training your ear. Adult language learners usually have brains that have become resistant to, but not incapable of, changing the parts of the b…

I completely agree with this. There's a certain confidence you get when you can hear a word you don't know, but can still comprehend it well enough to know what pinyin to type into your dictionary app. Mandarin Blueprint has a nice pinyin pronunciation video on YouTube that I worked through a while ago, and then followed with a few weeks of immersion in Taiwan, I was able to really pick out what people were saying.

I feel like listening is the key to speaking. You don't necessarily need to rote learn the tones for each word. You just need say words as you hear them spoken by others.

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#54

Post by dirteater_ »

et-al wrote: post #51
tifan wrote: post #40

Earlier quoted context omitted.

I had the same issue! Perhaps being another dapangzi is the problem here lol

I'm not familiar with this slang: what's a big plate?

the commenter's username (i'm guessing they mean 大胖子, feel free to google translate)

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#55

Post by dirteater_ »

cocoa19 wrote: post #48

Have you tried the Azure Speech Studio? I wonder how your custom model compares to this solution. I played around with python scripts for the same purpose. The AI gives feedback that can be transformed to a percentage of correctness. One annoyance is that for Mandarin, the percentage is calculated at the character level, whereas with English, it gives you a more granular score at the phoneme level.

IMO the SotA for this is https://www.speechsuper.com/. Amazon suffers for similar

> One annoyance is that for Mandarin, the percentage is calculated at the character level, whereas with English, it gives you a more granular score at the phoneme level.

This is the case for most solutions you'd find for this task. Probably because of the 1 character -> 1 syllable property. It's pretty straightforward to split the detected pinyin into initial+final and build a score from that though.

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#56

Post by jhanschoo »

sqs wrote: post #41
dapangzi wrote: post #13

Longtime lurker, made an account specifically to give feedback here as an intermediate speaker. :) This is a great initiative and I hope to see more come out of this; I am not criticizing, but just want to provide my user experience here so you have data points. In short, my experience lines up with your native speakers. I found that it loses track of the phonemes when speaking quickly, and tones don't seem to line u…

I don't think it takes care of tone transformation (eg 他是 ni3shi4 -> ni2shi4). Or if it does, my tones are just off. But it's a really cool idea!

The tone sandhi example you just gave looks incorrect to me

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#57

Post by barrell »

cyberax wrote: post #19
ecshafer wrote: post #15

Anyone that is a native European language speaker that hasn't tried to learn Chinese or some other tonal language, its really hard to understand how hard it is. The tones can really be very subtle, and your ear is not fine tuned to them. So you think you are saying it right, but native speakers have no idea what you are saying.

I'm a native Russian speaker, and I decided to learn Mandarin, because it's linguistically almost the opposite of Russian. I had no problems with tone pronunciation, but tone recognition was indeed much trickier. I still often get lost when listening to fast speech although I can follow formal speech (news) usually without problems.

I recently started learning a tonal language, and so far have not struggled too much wrt tones when everything is slow. There was an original strangeness and refusal for my vocal cords to want to work that way, but probably only for the first month or so.

At least, this is the case for slow text. Once the text is sped up it’s amazing how my brain just stops processing that information. Both listening and speaking.

I’m sure this will come with practice and time but for now I find it fascinating

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#58

Post by ronyeh »

laurieg wrote: post #28
ecshafer wrote: post #15

Anyone that is a native European language speaker that hasn't tried to learn Chinese or some other tonal language, its really hard to understand how hard it is. The tones can really be very subtle, and your ear is not fine tuned to them. So you think you are saying it right, but native speakers have no idea what you are saying.

For someone who hasn't grown up speaking an language with tones or pitches, the process of learning them can be maddening. I applaud anyone who makes tools like this to try to make the process easier. My experience in learning Japanese pitch accent was eye-opening. At the start, I couldn't hear any difference. On quizzes I essentially scored the same as random guessing. The first thing that helped me a lot was notici…

> For example, "uh-oh" has a high-low pitch. If you say it wrong it sounds very strange. "Uh-huh" to show understanding goes low-high. Again, if you reverse it it sounds unusual.

Wow… Thanks for making it clear that English also has tones! I hadn’t thought of it this way before. “Uh-huh” sounds similar to Mandarin tones 3 & 2. “Uh-oh” is similar to Cantonese tones 1 & 3.

I’m wondering if we can find good examples to teach the Mandarin tones. I think two or three syllable words are best because it illustrates the contour of the tones.

Re: Show HN: I trained a 9M speech model to fix my Mandarin tones

#60

Post by barrell »

bunderbunder wrote: post #45

This is very cool, but from one Mandarin learner to another I’d caution against relying too heavily on any external feedback mechanism for improving your pronunciation. If you can’t easily hear your pronunciation mistakes so clearly it hurts, consider putting more energy into training your ear. Adult language learners usually have brains that have become resistant to, but not incapable of, changing the parts of the b…

I’m building a language learning app [https://phrasing.app] and this is really good advice. I’ve not had any interest in SST for the application, and have no plans to integrate it. In my experience, I’ve never seen them be truly beneficial in the language learning process.

What has been extremely beneficial has been having the text and audio forced aligned and highlighted, kareoke-style, every time I hear the audio. It has improved my phoneme recognition remarkably well with remarkably little content. Several users also report the same thing - that even native speech feels a lot more like separate words than just a slew of sounds. I attribute this in large part just due to this kareoke style audio. It works better for phonetic scripts, so I would recommend using this with pinyin/jyutping/furigana for character based languages.

For production, when I was at Regina Coeli (world-class language institute) their main thing was just 1. you hear a short passage in Dutch, 10-40 words 2. you record yourself reading the same passage and 3. you play back the two audio tracks on top of one another and listen for the difference. Optional step 4. Re-record and replay until it’s close enough.

There was no grading, no teacher checking recordings, no right or wrong; just hundreds of random sentences and a simple app to layer them. You needed to learn to hear the differences yourself and experiment until you no longer could. (fwiw this is not present in phrasing, I just found it relevant. One day soon I hope to add it!)

Post reply on HN