Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sorting in Japanese – An Unsolved Problem (2011)

localizingjapan.com

Post reply on HN
Print view View original

91–100 of 120 posts

Re: Sorting in Japanese – An Unsolved Problem (2011)

#91

Post by SllX »

jayd16 wrote: post #71
SllX wrote: post #54

Earlier quoted context omitted.

I’ll back you up on this and add one additional point: even in the West where we have our Roman alphabet which largely works fine, and our Arabic numerals which also work fine, I’ve often thought adopting a limited number of kanji, literally less than 10, could add some important precision to our own writing. The rest of this post isn’t really for you specifically, but anyone reading through this comment thread with…

01d/01m/01y could be used just as well, no?

I’ve actually tried that and variations of that, and I’m not going to lie, it’s about as aesthetically pleasing to the eye as going back to Roman numerals.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#92

Post by expat2003 »

mopreme wrote: post #37

Blog author here. Surprised to see one of my old posts on the front page while browsing Hacker News. It's interesting to reflect on what has improved since I wrote it, and what has not. Both Android and iOS, for instance, provide mechanisms to get this right, if you know to use them and expose them for those locales (and only those locales). For example, both have a Contact object that contain corresponding phonetic-…

People doesn't seem to understand that they cannot have word-to-word translation when it comes to Japanese vs Latin root languages. Also there are grammatical and syntactical rules that you cannot get right by an educated guess - "educated guesses" such Google translate... See, Google translate doesn't care if a person doesn't know that when it comes to Proper names such as the name of a person the Kanji reads in a completely different way than the regular dictionary words. There is no translation for these class of words just as there is none for words like Jason, Tom, Samantha, etc. In such cases one needs to memorize the reading of the character, which reads that way only when it is used for Proper names. If the Kanji of a person have multiple reading such as 淳子, Atsuko, Kyoko and Junko, that person would use what is called Furigana, the Kana only reading showing the sound of that particular Kanji. This happens when you are filling a form, in a contract, in IDs etc. In other words feeding people names to Google only shows only one thing. That person doesn't know what is doing.

Sadako is Proper name of a popular iconic character in Japanese culture that embodies a little girl killer ghost and it is based on a real Japanese person that was said to be a psychic and lived in the 1900s. They made at least one horror Japanese movie out of that story and even Hollywood had a go at it! Now, how many girls you think they have been named Sadako in Japan by their parents? Once more Google translate doesn't give a damn if the person using it doesn't know what is doing.

In regular writing to mark Proper names a number of suffix are employed さま, どの, さん, くん, ちゃん and more. Every suffix has a different and unequivocal use case. It can be written in Hiragana, like I just did or by using Kanji. If the Kanji is adopted looks like 様 reads さま, 殿 reads どの etc. Also for such words there isn't a clear-cut translation, because they are employed as a result of the implications they come with. For instance 様 or さま is used when referring to customers or clients, but in a business letter you would use 殿 or どの with similar significance.

Most amusing is "淳子殿 translated to "Mr. Reiko". On Google Translate left windows in the left-bottom corner there is one of the possible correct reading of 淳子殿, namely Junko-dono. But on the right it's messing it right up. That's the demonstration that Google Translate has no brain. By the way Reiko is written 麗子 as a Proper name... and so is Rika, Akirako, Yoshiko, Urarako . Get the point? You need Furigana.

About gyoza, that's the name of a very popular kind of Chinese dumpling made in Japan, and there is no Kanji for it its written in Katakana only as チャオズ. But since it's a Chinese word you can used the two sounds ぎょう and ざ in Hiragana to write the two corresponding Kanji 餃子. To understand why, you need to read my post down the page. It seems Google translate cannot read the first Kanji right.

If you feel so inclined, mopreme, read my post down below which covers your original blog post extensively and throughout.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#93

Post by trw999 »

fenomas wrote: post #20
trw999 wrote: post #3

https://en.m.wikipedia.org/wiki/Four-Corner_Method You can sort Chinese characters (including Kanji but i'm not sure they use the Four Corners Method) by the Four Corners method. Why would you need to sort kanji phonetically in the first place? Do Japanese users actually expect names to be sorted phonetically? English speakers don't expect names to be sorted by IPA so consistency of the sorting scheme should be all t…

> Do Japanese users actually expect names to be sorted phonetically? Yes - or that's how a human would sort them, at any rate.

I don't think you're in a position to speak for all of humanity.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#94

Post by trw999 »

jwong_ wrote: post #17
trw999 wrote: post #3

https://en.m.wikipedia.org/wiki/Four-Corner_Method You can sort Chinese characters (including Kanji but i'm not sure they use the Four Corners Method) by the Four Corners method. Why would you need to sort kanji phonetically in the first place? Do Japanese users actually expect names to be sorted phonetically? English speakers don't expect names to be sorted by IPA so consistency of the sorting scheme should be all t…

Imagine having your contact list sorted by some geometric function run on each letter of the A-Z alphabet.

That would be marginally better than memorizing ABCDEFGHIJKMNOPQRSTUVWXYZ. Is there anything inherent about the letter A that makes it get sorted in front of the letter Z?

Re: Sorting in Japanese – An Unsolved Problem (2011)

#95

Post by Technetium_Hat »

tmm84 wrote: post #27

I've seen this problem solved a few different ways. 1) Have a romanized version of the value to sort. 2) Have a hiragana or katakana version of the value to sort (hiragana > katakana > roman order for sorting). Excel seems to sort things without telling it the reading for something. In JS and a few other languages Japanese is sorted based on UTF-8/16 code. This works for everything but kanji because of the reading re…

In JavaScript, I imagine using string.localeCompare would give a more helpful sort than just UTF-8 value.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#96

Post by Wowfunhappy »

fenomas wrote: post #73
Iv wrote: post #59

Earlier quoted context omitted.

You know, I started by loving kanjis too. I like the way they combine, the stories they tell. And I can imagine that when becoming proficient with them, they form a nice cozy system. But face it: you say I have years to learn them. I could learn 3 languages with the effort wasted to learn kanjis (which are half a language because then you also need to learn the pronunciation of words). No, I'll perfect my English ins…

I'm not sure it's even possible to get proficient at spoken Japanese without learning kanji, past some intermediate point. You either pick them up without intending to (enough to read anyway, if not write), or you don't get proficient. The issue being that proficiency in Japanese means navigating endless homophones and kanji compounds, and kanji are what disambiguates that process. Japanese vocabulary has a huge vari…

> I'm not sure it's even possible to get proficient at spoken Japanese without learning kanji, past some intermediate point.

Presumably the spoken Japanese language predates the writing system, no? Or if not, I would imagine that a couple hundred years ago, a significant portion of the population couldn't read Japanese but could speak it. How did they learn?

(Of course, let me know if both my assumptions are false—I don't know much about Japanese history beyond a single class in college, which focused on the modern era.)

Re: Sorting in Japanese – An Unsolved Problem (2011)

#97

Post by level3 »

For those who want to know more about this:

The article touches on just the tip of the iceberg. You might think that all you need to do is add an extra field for phonetic readings, and then simply sort on that field, but there are a lot of things that can go wrong. A naive sort (i.e. based simply on character code) will hit the following snags:

1) Hiragana vs Katakana

The article focuses more on Kanji vs Kana, but Japanese users will expect Hiragana and Katakana to be properly sorted together. Either you normalize your sort field (by converting everything to Hiragana, for example) or you use a Kana-insensitive collation.

2) Half-width characters

Katakana can be encoded as full-width or half-width characters (カ vs ｶ). Generally you want these treated as the same, so again you need to normalize or use a width-insensitive collation. There are also full-width alphabet characters (Ａ vs A).

3) Youon

These are actual different characters (ゆ vs ゅ, つ vs っ), so you can't normalize, but you want them sorted together. Here you need a collation that's case-insensitive with respect to these.

4) Dakuten/Handakuten

Like youon, these are also different characters (は vs ば vs ぱ) so you can't normalize, but you want them sorted together (insensitively). A sensitive sort will give you (はね, ばつ, ぱすた) while an insensitive sort will give you (ぱすた, ばつ, はね).

There has been a lot of work around this, resulting in many different database collations over the years, some of which result in sorts that would greatly confuse Japanese users. As of today, you probably want to be using (in the case of MySQL) utf8mb4_0900_ai_ci or something similar.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#98

Post by opan »

cmroanirgo wrote: post #80
needle0 wrote: post #69

As a Japanese native, it pains me a lot when the westerners come upon linguistic differences like this and start calling them things like "overly odd" or "inferior" as they seem to be doing in the comments here. I can't help but smell a whiff of anglophonic condescension - I had thought political correctness have swept over the west over the last few years, but perhaps they only selectively apply to those who can tal…

Reading the article showed me how ignorant I was to this problem... so much so that I'm surprised I haven't seen more articles like this in the past. For all the talk on ML and PC here, it's surprising that language is still a huge unsolved issue. As someoneone who knows a tiny amount of Japanese, I'm surprised that a common solution is to provide the katakana equivalent. Outside of the tech realm, is this really wha…

There was a scene in Sakurada Reset where the main character writes his name on a form in katakana and then is questioned about it. He says something about how people typically pronounce his name wrong so he writes it that way.

Re: Sorting in Japanese – An Unsolved Problem (2011)

#99

Post by GorgeRonde »

I think taking people names as an example is extreme because when Japanese people give a name to a baby, they choose both a phonological sign and a kanji-based transcription: phonologically, they tend to pick quite a common name (like christian names in the west, contrasting it with the native-american names that are a lot more specific to the person), but try to be clever and original when it comes to writing it with kanjis.

And I'm not even sure Amazon has an additional input field just for the sake of sorting names. Isn't this a common practice in the country ? (if I have to call a customer, how am I supposed to greet her if I can't pronounce her name ?)

Re: Sorting in Japanese – An Unsolved Problem (2011)

#100

Post by bobthepanda »

segfaultbuserr wrote: post #23
bobthepanda wrote: post #19

Earlier quoted context omitted.

Based on extremely limited Googling, one of the cases where these codes are still used is written colloquial Cantonese, which lacks any major official support.

Good point. Some forms of Cantonese romanization ( https://en.wikipedia.org/wiki/Hong_Kong_Government_Cantonese... ) does exist, but their use is limited to mainly language study and transliteration. And apparently, there are multiple projects to create phonetic input systems for Cantonese (see the reference section of this Cantonese Wikipedia article, https://zh-yue.wikipedia.org/wiki/%E7%B2%B5%E8%AA%9E%E6%8B%B... )…

As someone who has attempted to learn to get back to my roots, I can definitely say the lack of a standard romanization is a huge barrier. Cantonese textbooks are not interchangeable because each series decides which romanization to use and some even take the liberty of creating their own.

The only people that could sort this out is a government.

- Guangdong won’t, because official Party policy at best discourages use of regional languages. There was a lot of fury back when they attempted to stop provincial broadcasts in Cantonese.

- Hong Kong won’t, because the Government likes to fumble around a lot these days, and because they have a more or less unofficial goal to integrate into China. Cantonese is not an official language, only “Chinese”.

- Macau won’t. They’re too busy trying to sweeten up China to let in more gambling tourists.

The other issue is that there are now some phonetic changes between the mainland and Hong Kong, the two places people would consider authoritative on the subject.

Post reply on HN