Projecting Unicode to ASCII • Hacker News Board

Re: Projecting Unicode to ASCII

#21

Post by lotyrin » Thu, Jan 10, 2019, 4:43 PM UTC

avian wrote: post #12

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Unidecode maintainer here. I really need to add some commentary to that "Bei Jing" example in the README. Unidecode doesn't do language-specific transliteration and really works best for user-invisible things, like database identifiers or normalization. CJK characters in particular are very problematic, since they must be transliterated differently depending on the locale. Over the years I have received many angry ma…

One of those angry users here -- e.g. making flash card decks for Japanese on Memrise where they generate URLs based on some choice set of Chinese transliterations for no discernible reason.

Re: Projecting Unicode to ASCII

#22

Post by mark-r » Thu, Jan 10, 2019, 4:52 PM UTC

The formal definition of "project" here doesn't mesh well with the way it's commonly used in English. You don't double apply a projection simply because it doesn't make any sense to try. Think of a film projector or a Mercator map.

Re: Projecting Unicode to ASCII

#23

Post by gumby » Thu, Jan 10, 2019, 5:54 PM UTC

kuschku wrote: post #9

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Unidecode may handle Chinese fine, but it definitely handles Western European languages wrong. DIN 5007 Var. 2 specifies that for the purposes of sorting, ö is replaced with oe. This also applies to ä (ae), ü (ue) and ß (ss). This same replacement rule is also used on passports and IDs. Unidecode does not handle this correctly. The unidecode author wrote about this: > In German, there's the typographical convention t…

I agree with this -- simply dropping the umlaut is adequately clear, weird as it may look to me, just as dropping a circumflex is preferred to adding an 's' after it in French. Adding an e doesn't produce a reversible result (there are, for example, ordinary German words written with ae, oe, and ss).

Every such projection is a lossy projection and is expected to possibly generate in an ambiguous result that must be interpreted through context. Hence the strange but comprehensible ,,Godel'' and ,,Malmo''. Cook is not claiming that his code replaces the need for Unicode!

(There is a minor linguistic irony that the origin of the umlaut was scribe's shorthand when an E vowel inflection was turned into tiny E written above a letter, almost like a ligature, which became a pair of dots. But that character was then used in other languages differently, much as a loanword from a different language usually changes its meaning in the new language).

Re: Projecting Unicode to ASCII

#24

Post by gumby » Thu, Jan 10, 2019, 6:04 PM UTC

aaaaaaaaaab wrote: post #20

I use this transform chain for asciifying filenames (ridiculous as it is, in 2019 we still can't sync unicode filenames between different OSs): uconv -x ':: Any-Latin; :: Latin-ASCII; [:^ASCII:] > \_' Explanation: Any-Latin Transliterates from non-latin scripts to latin script (e.g. "γραφὴν" --> "graphḕn"). Latin-ASCII Tries to asciify characters as much as possible by discarding accents, splitting ligatures, replaci…

Why not transform them into utf-8, say with iconv?

Re: Projecting Unicode to ASCII

#25

Post by umlautae » Thu, Jan 10, 2019, 6:06 PM UTC

We are actually actively working on Cyrillic-ASCII transliteration for glibc in this very moment. Please check this patch for details [1] Your help and suggestions welcome to make sure this will be a useful and consistent fix when it lands. The bug [2] is from 2006 (sic!) and it is a reason why transliteration may not work with iconv for some systems/locales. [1] https://sourceware.org/ml/libc-locales/2019-q1/msg00010.html [2]https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Re: Projecting Unicode to ASCII

#26

Post by gumby » Thu, Jan 10, 2019, 6:07 PM UTC

iconv is the standard utility for this (a standard unix/linux utility) and includes the projection of characters that aren't in the target character set.

Not sure how well it handles CJKV chars though.

Re: Projecting Unicode to ASCII

#27

Post by umlautae » Thu, Jan 10, 2019, 6:13 PM UTC

aaaaaaaaaab wrote: post #20

I use this transform chain for asciifying filenames (ridiculous as it is, in 2019 we still can't sync unicode filenames between different OSs): uconv -x ':: Any-Latin; :: Latin-ASCII; [:^ASCII:] > \_' Explanation: Any-Latin Transliterates from non-latin scripts to latin script (e.g. "γραφὴν" --> "graphḕn"). Latin-ASCII Tries to asciify characters as much as possible by discarding accents, splitting ligatures, replaci…

Do you care to chime in with your use case for our patch discussion? I am trying to argue for the translit to be most useful for asciifying filenames among the others. Another actual user will help to make the case. https://sourceware.org/ml/libc-locales/2019-q1/msg00014.html

Re: Projecting Unicode to ASCII

#28

Post by akie » Thu, Jan 10, 2019, 6:22 PM UTC

mci wrote: post #13

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Fredrik Lundh describes a DIY patch on top of unicodedata here: http://effbot.org/zone/unicode-convert.htm It solves the ä->ae problem and its ilk for Western European languages. Time has devoured my comment on that post, which extended the transliteration table to Eastern European languages and proposed to use mnemonic names like int(u'\N{Latin capital letter AE}') instead of 0xc6.

ä -> ae is a German transliteration that would not be recognized by a non-German (e.g. a Dutchman).

Re: Projecting Unicode to ASCII

#29

Post by johndcook » Thu, Jan 10, 2019, 7:08 PM UTC

avian wrote: post #12

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Unidecode maintainer here. I really need to add some commentary to that "Bei Jing" example in the README. Unidecode doesn't do language-specific transliteration and really works best for user-invisible things, like database identifiers or normalization. CJK characters in particular are very problematic, since they must be transliterated differently depending on the locale. Over the years I have received many angry ma…

Thank you for maintaining this package. I wish I'd run across it sooner.

Re: Projecting Unicode to ASCII

#30

Post by waynecochran » Thu, Jan 10, 2019, 7:26 PM UTC

vesinisa wrote: post #11

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

> if you need to project (transliterate to ascii) Chinese unidecode is close to black magic That might be a dangerous assumption. The project page explicitly states: > Transliteration of languages like Chinese is a very complex issue and this library does not even attempt to address it. It draws the line at context-free character-by-character mapping. I.e. it is not black magic; just a mapping of Unicode characters t…

Perhaps for some applications, e.g. hash keys, consistency is more important than correctness.