Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sorting in Japanese – An Unsolved Problem (2011)

localizingjapan.com

Post reply on HN
Print view View original

21–30 of 120 posts

Re: Sorting in Japanese – An Unsolved Problem (2011)

#21

Post by segfaultbuserr »

jwong_ wrote: post #18
segfaultbuserr wrote: post #14

Earlier quoted context omitted.

Don't know about Japanese, but other well-known input schemes for Chinese includes Cangjie ( https://en.wikipedia.org/wiki/Cangjie_input_method ), Zhengma ( https://en.wikipedia.org/wiki/Zhengma_method ), and Wubi ( https://en.wikipedia.org/wiki/Wubi_method ). In fact, all of these non-phonetic input/encoding systems are highly non-intuitive and have a reputation of sharp learning curves, frustration is expected. Thi…

Phonetic systems for Chinese aren't so prolific for non-mandarin speakers. 9 square/Q9 is another popular method. The stroke-based methods aren't so arbitrary - they are based on the way you write.

> The stroke-based methods aren't so arbitrary - they are based on the way you write.

Indeed, they are not arbitrary.

The very invention of them is meant to create something much more meaningful to the users as alternative to the telegraph code and alike, which is nothing more than a bunch of numbers. But I say "arbitrary", in the sense that the classification and organization of strokes in the system is selected artificially by the designers, not something inherently exists in the language and understood by the speakers, difficulties are overcame once you are familiar to the system. But extensive effort is needed for a new user to master the system.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#22

Post by kijin »

aliswe wrote: post #2

If im not mistaken, another unsolved problem is generating url segments (or "slugs") from characters such as chinese, arabic and possibly japanese as well.

You just use a (somewhat cleaned up) UTF-8 representation as a slug. UTF-8 URL components are universally supported (transparently encoded and decoded) in all modern browsers, and Google shows the real characters instead of the urlencoded version. You still end up with the urlencoded version when you hit Ctrl+C, but I expect that to be fixed in the near future as Unicode becomes even more widely used.

For example, the following link should work perfectly well in all modern browsers:

https://ja.wikipedia.org/wiki/メインページ

Re: Sorting in Japanese – An Unsolved Problem (2011)

#23

Post by segfaultbuserr »

bobthepanda wrote: post #19
segfaultbuserr wrote: post #14

Earlier quoted context omitted.

Don't know about Japanese, but other well-known input schemes for Chinese includes Cangjie ( https://en.wikipedia.org/wiki/Cangjie_input_method ), Zhengma ( https://en.wikipedia.org/wiki/Zhengma_method ), and Wubi ( https://en.wikipedia.org/wiki/Wubi_method ). In fact, all of these non-phonetic input/encoding systems are highly non-intuitive and have a reputation of sharp learning curves, frustration is expected. Thi…

Based on extremely limited Googling, one of the cases where these codes are still used is written colloquial Cantonese, which lacks any major official support.

Good point.

Some forms of Cantonese romanization (https://en.wikipedia.org/wiki/Hong_Kong_Government_Cantonese...) does exist, but their use is limited to mainly language study and transliteration. And apparently, there are multiple projects to create phonetic input systems for Cantonese (see the reference section of this Cantonese Wikipedia article, https://zh-yue.wikipedia.org/wiki/%E7%B2%B5%E8%AA%9E%E6%8B%B...), but all with very limited standardization and official supports.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#24

Post by mjevans »

cooper12 wrote: post #5

Just a thought experiment, don't take it too seriously: The crux of the issue is that kanji don't have an inherent "natural" ordering that a user would expect. Sorting by their character code doesn't mean anything to a Japanese person. But, what if we made our own standard of what entails a "natural order". There's nothing about A–Z that makes the alphabet obligated to be in that order (and not something like based o…

The concept of having a displayed value and a value to actually sort by isn't limited to Japanese words/names. It also comes up when there's a desired display function but for sorting intents different parts should be added or removed from the name of an entity. One such example is book and movie titles in a library.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#25

Post by innocenat »

cooper12 wrote: post #5

Just a thought experiment, don't take it too seriously: The crux of the issue is that kanji don't have an inherent "natural" ordering that a user would expect. Sorting by their character code doesn't mean anything to a Japanese person. But, what if we made our own standard of what entails a "natural order". There's nothing about A–Z that makes the alphabet obligated to be in that order (and not something like based o…

> Even hiragana can have different orderings (AIUEO vs IROHA [0])

Unless otherwise mentioned, things are almost always AIUEO nowadays.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#26

Post by csours »

asutekku wrote: post #9
hermitdev wrote: post #4

Am I missing something subtle in this Kanji example, or all 4 names actually written the same?: "There are four Japanese women whose names you have to sort: Junko, Atsuko, Kiyoko, and Akiko. This does not seem difficult, until they each show you how they write their names in kanji: 淳子 (Junko) 淳子 (Atsuko) 淳子 (Kiyoko) 淳子 (Akiko)" I'm not familiar with Japanese at all (and have never had to deal with localization beyond…

That’s why japanese people sometimes when introducing themselves also mention the kanjis because of the wildly different readings. There is literally no way knowing the reading if the user doesn’t provide it.

Is this why business cards are so ubiquitous?

Re: Sorting in Japanese – An Unsolved Problem (2011)

#27

Post by tmm84 »

I've seen this problem solved a few different ways.

1) Have a romanized version of the value to sort. 2) Have a hiragana or katakana version of the value to sort (hiragana > katakana > roman order for sorting).

Excel seems to sort things without telling it the reading for something. In JS and a few other languages Japanese is sorted based on UTF-8/16 code. This works for everything but kanji because of the reading requires a human.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#28

Post by rococode »

hermitdev wrote: post #4

Am I missing something subtle in this Kanji example, or all 4 names actually written the same?: "There are four Japanese women whose names you have to sort: Junko, Atsuko, Kiyoko, and Akiko. This does not seem difficult, until they each show you how they write their names in kanji: 淳子 (Junko) 淳子 (Atsuko) 淳子 (Kiyoko) 淳子 (Akiko)" I'm not familiar with Japanese at all (and have never had to deal with localization beyond…

Many kanji have numerous pronunciations. I have a dictionary extension on my browser that may help illustrate:

https://i.imgur.com/Nftkjks.png

According to the dictionary, the word 淳 can be read as jun, shun, atsui, atsu, atsushi, kiyo, kiyoshi, makoto, or sunao. And evidently this is not fully complete, since Akiko can also use that character.

It's definitely one of the more annoying parts of Japanese, and ultimately you just have to learn common words and for ones where there are multiple common pronunciations (like these names), someone will ask for the pronunciation (for names) or it will be clear from context (for words).

Also, if you weren't aware, there's also a thing called furigana, which is essentially writing the alphabet really small above a kanji word so you know the pronunciation. Really useful and something that Chinese sorely lacks imo.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#29

Post by jpatokal »

cooper12 wrote: post #5

Just a thought experiment, don't take it too seriously: The crux of the issue is that kanji don't have an inherent "natural" ordering that a user would expect. Sorting by their character code doesn't mean anything to a Japanese person. But, what if we made our own standard of what entails a "natural order". There's nothing about A–Z that makes the alphabet obligated to be in that order (and not something like based o…

The only problem with that is that the resulting order would be useless for many applications. Say you're looking up your friend Tanaka Tarou, but you're not sure which characters his name is written with. If the sort order is phonetic, you can find the name and likely work out that this is the Tanaka you were looking for. But how do you search for a name in a kanji-indexed list if you don't know the kanji?

Incidentally, this is why kanji dictionaries invariably have multiple indices: one by radical, the other by pronunciation(s).

Re: Sorting in Japanese – An Unsolved Problem (2011)

#30

Post by cooper12 »

jpatokal wrote: post #29
cooper12 wrote: post #5

Just a thought experiment, don't take it too seriously: The crux of the issue is that kanji don't have an inherent "natural" ordering that a user would expect. Sorting by their character code doesn't mean anything to a Japanese person. But, what if we made our own standard of what entails a "natural order". There's nothing about A–Z that makes the alphabet obligated to be in that order (and not something like based o…

The only problem with that is that the resulting order would be useless for many applications. Say you're looking up your friend Tanaka Tarou, but you're not sure which characters his name is written with. If the sort order is phonetic, you can find the name and likely work out that this is the Tanaka you were looking for. But how do you search for a name in a kanji-indexed list if you don't know the kanji? Incidenta…

Great point. I was considering a very visual-minded reader but of course not everyone would be so good at it nor would they always just care about how the kanji looks rather than other aspects. It's a difficult problem indeed... My intention is to show that we'd need some sort of radical solution that might not be what we'd immediately jump to (for example the current approach the author mentions is having a separate field for readings, but this is clearly resource-intensive and wouldn't work on arbitrary data). To solve sorting for Japanese, I feel we need to rethink what it means to sort.
Post reply on HN