Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Neural Japanese Transliteration

github.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Show HN: Neural Japanese Transliteration

#41

Post by ekianjo »

hasenj wrote: post #39

I'm more interested in (kind of) the reverse. Given a Japanese sentence (that uses kanji), figure out the proper reading for each Kanji character, using a neutral network. I know there are already hardcoded analyzers, like kuromoji, but they produce incorrect answers in a lot of edge cases.

its very hard to do that since there are cases when you can read the kanji In multiple ways, like In peoples names for examples. Japanese is full of exceptions because the writing system was imported very, very late In Japan (300ad) without much effort to standardize its application.

Re: Show HN: Neural Japanese Transliteration

#43

Post by ekianjo »

derefr wrote: post #12
microcolonel wrote: post #6

Earlier quoted context omitted.

For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel.…

Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name. If CJK IMEs expected and re…

pause between each word? certainly not. Japanese are known to speak very fast and to remove any form of blank between words. like when asking お元気ですか? it sounds as if everything is attached. Please dont spread out inaccurate information.

Re: Show HN: Neural Japanese Transliteration

#44

Post by GolDDranks »

derefr wrote: post #12
microcolonel wrote: post #6

Earlier quoted context omitted.

For what it's worth, this is basically just the same as any popular Japanese (or Chinese) input method. Usually the approach is to greedily form the smallest set of the longest words from the given syllables, because people tend to give inputs where all the words are complete. Sometimes people use markov models to fix situations where that falls over. Not sure how well this model performs, but the task is not novel.…

Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name. If CJK IMEs expected and re…

There isn't certainly pauses between words in Japanese. (Actually, I'm not aware of any language that has.)

The Japanese disambiguate word boundaries in spoken language using the pitch accent as the primary clue. Tokyo Japanese has a phenomenon called initial rise, which differentiates the pitch between the two first moras of an accent phrase – either the pitch rises or steeply falls.

Here's an example - upper case: high pitch, lower case: low pitch.

  KYOu,  kaINI  iKIMAshita
  today, to buy I went
  
  KYOu   KAini      iKIMAshita
  today, to meeting I went
  
  kyoUKAINI iKIMAshita
  to chuckh I went

Re: Show HN: Neural Japanese Transliteration

#45

Post by alphonsegaston »

evincarofautumn wrote: post #36
kobeya wrote: post #31

Earlier quoted context omitted.

> Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else. Also intonation, which is not captured by the written system at all. Japanese isn't strongly tonal in the way Chinese is, but it has a regional prosody, like Swedish, which helps in disambiguating meaning.

It’s usually analysed as having both normal prosody and a pitch accent (similar to a stress accent) that varies somewhat by region. I’ve read that broadcasters are expected to use a standard (Tokyo?) pitch accent when speaking.

That is correct. The Tokyo dialect is considered "neutral" in the same way a Midwestern accent is for American broadcasters.

Re: Show HN: Neural Japanese Transliteration

#47

Post by rspeer »

ekianjo wrote: post #41
hasenj wrote: post #39

I'm more interested in (kind of) the reverse. Given a Japanese sentence (that uses kanji), figure out the proper reading for each Kanji character, using a neutral network. I know there are already hardcoded analyzers, like kuromoji, but they produce incorrect answers in a lot of edge cases.

its very hard to do that since there are cases when you can read the kanji In multiple ways, like In peoples names for examples. Japanese is full of exceptions because the writing system was imported very, very late In Japan (300ad) without much effort to standardize its application.

No need to shoot down an NLP task because it can't be solved with 100% accuracy. That's every NLP task.

The kanji -> kana direction should be considerably easier than the kana -> kanji direction. There are many fewer sources of ambiguity, and the space of possible answers is smaller.

Re: Show HN: Neural Japanese Transliteration

#48

Post by GolDDranks »

ekianjo wrote: post #42
scentedmeat wrote: post #34

> In the digital environment, people mostly type Roman alphabet Might be selection bias but I mostly notice people using the 10-key click one

except in asia. Koreans and Japanese use a lot of dial type input systems.

Not sure what you mean by dial type input system, but if you mean that people use the number keys 0-9 to input text, that's the exact thing the grantparent meant.

Re: Show HN: Neural Japanese Transliteration

#49

Post by ue_ »

Iv wrote: post #40
scentedmeat wrote: post #34

> In the digital environment, people mostly type Roman alphabet Might be selection bias but I mostly notice people using the 10-key click one

True on the smartphone. On the computer most Japanese speakers I know just type romaji. However, this is pretty much irrelevant to this article, as romaji->kanas (the phonetic alphabets) is a pretty straightforward and solved problem (there is a clear bijection between both). The real problem is transforming the phonetic transliteration into the correct word in either kanji (for most Japanese words) or katakana (for…

Do people really use romaji on keyboards in Japan? This strikes me as an odd way to type, as it means that you first need to learn romaji in order to type. I thought that hiragana keyboards (hiragana mapped onto the normal layout) were the norm, especially on laptop keyboards.
Post reply on HN