Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Neural Japanese Transliteration

github.com

Post reply on HN
Print view View original

11–20 of 55 posts

Re: Show HN: Neural Japanese Transliteration

#11

Post by alphonsegaston »

blipmusic wrote: post #7
alphonsegaston wrote: post #4

Interesting project! Does anyone know what iOS is using for it Japanese transliteration predictions? Mine worked well for a long time, but in the past 3 months it's gone haywire for common kanji suggestions. The other day it had "機能" as the first/only suggestion for "きのう," and I had to dig down into the menu to arrive at the intended, "昨日." Had a lot of similar experiences recently.

Does it act up on your desktop OS as well? I know there's an alternative from the ATOK developers for iOS ( http://www.justsystems.com/jp/products/atok_ios/ ) but I haven't tried that one.

No, just my phone. But I didn't know about that alternative. Thanks, I'll check it out!

Re: Show HN: Neural Japanese Transliteration

#12

Post by derefr »

microcolonel wrote: post #6
blipmusic wrote: post #3

Cool! So, would it be correct to say that this is in essence generates a disambiguation model for a language with a lot of homonyms due to having relatively few sounds (but also with some variation in morpheme boundaries, e.g. "an-i" vs "ani")?

For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel.…

Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else.

But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name.

If CJK IMEs expected and required people to hit the spacebar between the "words" (lexer tokens) of the provided input for matching, they'd have a much simpler job. But as it is, they fall over quite badly when you type multiple words into the IME input box, and are mostly only usable if you resolve single words at a time (which, sadly, throws away a lot of the inter-word context that would otherwise be available for matching.)

Re: Show HN: Neural Japanese Transliteration

#13

Post by derefr »

microcolonel wrote: post #10
blipmusic wrote: post #9

Earlier quoted context omitted.

Well, to be honest my first impression was that this is what I've been using all along. I just liked that someone experimented a bit. :) Perhaps I assumed this would end up with a different predictive model. No idea what kind of corpora have been used for the input methods already available. Edit: your P.S. made me remember the "ma-ma-ma-ma..." mouthful the Chinese language students I studied in parallel with discove…

For what it's worth, there's a lot of work to do on Japanese input methods. To a human it may be obvious that hawotogu should become 刃を研ぐ (sharpen a/the blade), and that one does not typically sharpen or hone a 歯 (tooth), 葉(leaf), 派(party/ingroup), or even 覇(~hegemony); but to a computer without topic-specific context, each of these is equally valid. One might be thankful that humans don't really say all that many di…

I don't know about "topic-specific context", but doesn't a word-granular Hidden Markov Model trained on huge amounts of text from the language let you figure out things like "given that the sentence includes 刃, 研 is the highest-scoring match"?

Re: Show HN: Neural Japanese Transliteration

#14

Post by gizmo686 »

derefr wrote: post #12
microcolonel wrote: post #6

Earlier quoted context omitted.

For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel.…

Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name. If CJK IMEs expected and re…

>because Japanese people pause between spoken words just like anyone else.

This is (surprisingly) not true. People do not pause between words, however when listening to a language that they understand, they do perceive pauses between words; even though such pauses do not exist.

Re: Show HN: Neural Japanese Transliteration

#15

Post by microcolonel »

derefr wrote: post #13
microcolonel wrote: post #10

Earlier quoted context omitted.

For what it's worth, there's a lot of work to do on Japanese input methods. To a human it may be obvious that hawotogu should become 刃を研ぐ (sharpen a/the blade), and that one does not typically sharpen or hone a 歯 (tooth), 葉(leaf), 派(party/ingroup), or even 覇(~hegemony); but to a computer without topic-specific context, each of these is equally valid. One might be thankful that humans don't really say all that many di…

I don't know about "topic-specific context", but doesn't a word-granular Hidden Markov Model trained on huge amounts of text from the language let you figure out things like "given that the sentence includes 刃, 研 is the highest-scoring match"?

It does, but the suggestions will be nowhere to be found if somebody is making a novel use of words. That's what I was getting at.

Re: Show HN: Neural Japanese Transliteration

#16

Post by yborg »

alphonsegaston wrote: post #4

Interesting project! Does anyone know what iOS is using for it Japanese transliteration predictions? Mine worked well for a long time, but in the past 3 months it's gone haywire for common kanji suggestions. The other day it had "機能" as the first/only suggestion for "きのう," and I had to dig down into the menu to arrive at the intended, "昨日." Had a lot of similar experiences recently.

I've had a similar experience on OS X, it seems to lose its mind and actually refuse to display as a choice the most common version of a jukugo. Very weird.

These is also a reset for "Conversion Learning" under the Keyboard > Input Sources > Japanese that I've just found thanks to the hints from the thread on this feature in iOS.

Re: Show HN: Neural Japanese Transliteration

#17

Post by schoen »

blipmusic wrote: post #9
microcolonel wrote: post #6

Earlier quoted context omitted.

For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel.…

Well, to be honest my first impression was that this is what I've been using all along. I just liked that someone experimented a bit. :) Perhaps I assumed this would end up with a different predictive model. No idea what kind of corpora have been used for the input methods already available. Edit: your P.S. made me remember the "ma-ma-ma-ma..." mouthful the Chinese language students I studied in parallel with discove…

I don't speak Chinese but I've encountered this series a lot:

https://en.wiktionary.org/wiki/%E5%AA%BD#Chinese 媽 mā 'mother'

https://en.wiktionary.org/wiki/%E9%BA%BB#Chinese 麻 má 'hemp' (sometimes 'flax')

https://en.wiktionary.org/wiki/%E9%A6%AC#Chinese 馬 mǎ 'horse'

https://en.wiktionary.org/wiki/%E7%BD%B5#Chinese 罵 mà 'scold'

It's also cool that you can see that 媽 is made up of "semantic 女 + phonetic 馬", where 女 means 'lady' and 馬 sounds like "ma", so the character was meant to suggest "a word relating to ladies that sounds like ma".

https://en.wikipedia.org/wiki/Chinese_character_classificati...

Re: Show HN: Neural Japanese Transliteration

#18

Post by blipmusic »

microcolonel wrote: post #10
blipmusic wrote: post #9

Earlier quoted context omitted.

Well, to be honest my first impression was that this is what I've been using all along. I just liked that someone experimented a bit. :) Perhaps I assumed this would end up with a different predictive model. No idea what kind of corpora have been used for the input methods already available. Edit: your P.S. made me remember the "ma-ma-ma-ma..." mouthful the Chinese language students I studied in parallel with discove…

For what it's worth, there's a lot of work to do on Japanese input methods. To a human it may be obvious that hawotogu should become 刃を研ぐ (sharpen a/the blade), and that one does not typically sharpen or hone a 歯 (tooth), 葉(leaf), 派(party/ingroup), or even 覇(~hegemony); but to a computer without topic-specific context, each of these is equally valid. One might be thankful that humans don't really say all that many di…

I didn't mean to imply otherwise. My curiosity was/is with the generation of the model in the link (especially compared to current models), hence the question in my original post. There's also pitch accent that helps with disambiguation of spoken Japanese (I should know, I accidentally called a friend seaweed for a long time).

Re: Show HN: Neural Japanese Transliteration

#19

Post by PeCaN »

schoen wrote: post #17
blipmusic wrote: post #9

Earlier quoted context omitted.

Well, to be honest my first impression was that this is what I've been using all along. I just liked that someone experimented a bit. :) Perhaps I assumed this would end up with a different predictive model. No idea what kind of corpora have been used for the input methods already available. Edit: your P.S. made me remember the "ma-ma-ma-ma..." mouthful the Chinese language students I studied in parallel with discove…

I don't speak Chinese but I've encountered this series a lot: https://en.wiktionary.org/wiki/%E5%AA%BD#Chinese 媽 mā 'mother' https://en.wiktionary.org/wiki/%E9%BA%BB#Chinese 麻 má 'hemp' (sometimes 'flax') https://en.wiktionary.org/wiki/%E9%A6%AC#Chinese 馬 mǎ 'horse' https://en.wiktionary.org/wiki/%E7%BD%B5#Chinese 罵 mà 'scold' It's also cool that you can see that 媽 is made up of "semantic 女 + phonetic 馬", where 女 mea…

There's also this rather fun poem, Shī Shì shí shī shǐ.

https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_...

Post reply on HN