Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Neural Japanese Transliteration

github.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Show HN: Neural Japanese Transliteration

#31

Post by kobeya »

derefr wrote: post #12
microcolonel wrote: post #6

Earlier quoted context omitted.

For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel.…

Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name. If CJK IMEs expected and re…

> Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else.

Also intonation, which is not captured by the written system at all. Japanese isn't strongly tonal in the way Chinese is, but it has a regional prosody, like Swedish, which helps in disambiguating meaning.

Re: Show HN: Neural Japanese Transliteration

#32

Post by minikomi »

alphonsegaston wrote: post #4

Interesting project! Does anyone know what iOS is using for it Japanese transliteration predictions? Mine worked well for a long time, but in the past 3 months it's gone haywire for common kanji suggestions. The other day it had "機能" as the first/only suggestion for "きのう," and I had to dig down into the menu to arrive at the intended, "昨日." Had a lot of similar experiences recently.

I also had this problem and had to turn off "Live Conversion" in the Japanese settings.

Re: Show HN: Neural Japanese Transliteration

#35

Post by gizmo686 »

gizmo686 wrote: post #30
snakeboy wrote: post #25

Earlier quoted context omitted.

It seems like it'd be useful for one or both of you to cite any research than has been done on this. Seems more productive (and enlightening to all) than the agree/disagree dialogue here.

Unfourtuantly, this is so well established that it is hard to find research. I did find this paper [1] which looks into how babies acuire word boundries. As you identify, there are probably phonetic cues; but not pauses. The best way to see this is to try listening to a language you do not understand, and try to identify word boundries. Indeed, the paper I link argues that some phonetic cue must exist because babies…

It seems I linked only to the abstract. Here [1] is the pdf.

[1] https://www.sissa.it/cns/Articles/94_doInfPerceiveWordB.pdf

Re: Show HN: Neural Japanese Transliteration

#36

Post by evincarofautumn »

kobeya wrote: post #31
derefr wrote: post #12

Earlier quoted context omitted.

Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name. If CJK IMEs expected and re…

> Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. Also intonation, which is not captured by the written system at all. Japanese isn't strongly tonal in the way Chinese is, but it has a regional prosody, like Swedish, which helps in disambiguating meaning.

It’s usually analysed as having both normal prosody and a pitch accent (similar to a stress accent) that varies somewhat by region. I’ve read that broadcasters are expected to use a standard (Tokyo?) pitch accent when speaking.

Re: Show HN: Neural Japanese Transliteration

#37

Post by shawnps »

scentedmeat wrote: post #34

> In the digital environment, people mostly type Roman alphabet Might be selection bias but I mostly notice people using the 10-key click one

I agree. I use it on iOS with swiping enabled - it is way faster and less error-prone than the romaji keyboard.

Re: Show HN: Neural Japanese Transliteration

#38

Post by fenomas »

microcolonel wrote: post #6
blipmusic wrote: post #3

Cool! So, would it be correct to say that this is in essence generates a disambiguation model for a language with a lot of homonyms due to having relatively few sounds (but also with some variation in morpheme boundaries, e.g. "an-i" vs "ani")?

For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel.…

> Japanese, to my elementary-level ear, doesn't sound an order of magnitude more ambiguous than English

I don't have any citations, but subjectively I disagree. One can think of English words that are homophones, but in JP the challenge is more to think of words that aren't.

On one hand, if you include uncommon words (as a keyboard's corpus would) then practically any medium-length word will have multiple possibilities, which is not the case in English. But much more important is that, where an English homophone typically has two or possibly three interpretations, a kanji jukugo might have 3-4 everyday meanings and a bunch more uncommon ones.

And all this is on top of the matter of word boundaries. If the user enters し, there could be 10+ possible transliterations of that character as a standalone word/particle, above and beyond whatever readings are possible together with the characters before and after.

So I don't know what an order of magnitude would mean in this context, but I think the whole matter is significantly more ambiguous than English.

Re: Show HN: Neural Japanese Transliteration

#39

Post by hasenj »

I'm more interested in (kind of) the reverse.

Given a Japanese sentence (that uses kanji), figure out the proper reading for each Kanji character, using a neutral network.

I know there are already hardcoded analyzers, like kuromoji, but they produce incorrect answers in a lot of edge cases.

Re: Show HN: Neural Japanese Transliteration

#40

Post by Iv »

scentedmeat wrote: post #34

> In the digital environment, people mostly type Roman alphabet Might be selection bias but I mostly notice people using the 10-key click one

True on the smartphone. On the computer most Japanese speakers I know just type romaji. However, this is pretty much irrelevant to this article, as romaji->kanas (the phonetic alphabets) is a pretty straightforward and solved problem (there is a clear bijection between both).

The real problem is transforming the phonetic transliteration into the correct word in either kanji (for most Japanese words) or katakana (for words with foreign origin).

This problem is akin to disambiguating between two homonymes (which are much more frequent in Japanese). In some cases it is easy by looking the previous words, but in some it is heavily context dependent.

Nowadays, most japanese typing system will propose a list of kanjis as you type that corresponds to the most frequent writting of your transliteration, but sometimes for unusual kanjis o(or people's name) you have to dig deep into the list.

I can see how such a system could improve typing speed in Japanese.

Post reply on HN