Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Projecting Unicode to ASCII

johndcook.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Projecting Unicode to ASCII

#11

Post by vesinisa »

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

> if you need to project (transliterate to ascii) Chinese unidecode is close to black magic

That might be a dangerous assumption. The project page explicitly states:

> Transliteration of languages like Chinese is a very complex issue and this library does not even attempt to address it. It draws the line at context-free character-by-character mapping.

I.e. it is not black magic; just a mapping of Unicode characters to static ASCII transliterations. The results will certainly be incorrect in some contexts.

Re: Projecting Unicode to ASCII

#12

Post by avian »

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Unidecode maintainer here. I really need to add some commentary to that "Bei Jing" example in the README.

Unidecode doesn't do language-specific transliteration and really works best for user-invisible things, like database identifiers or normalization.

CJK characters in particular are very problematic, since they must be transliterated differently depending on the locale. Over the years I have received many angry mails from people that were deeply offended by an error in transliteration they saw in an URL or something.

Unihandecode is a fork of Unidecode that tries to address this:

https://github.com/miurahr/unihandecode

Re: Projecting Unicode to ASCII

#13

Post by mci »

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Fredrik Lundh describes a DIY patch on top of unicodedata here: http://effbot.org/zone/unicode-convert.htm It solves the ä->ae problem and its ilk for Western European languages.

Time has devoured my comment on that post, which extended the transliteration table to Eastern European languages and proposed to use mnemonic names like int(u'\N{Latin capital letter AE}') instead of 0xc6.

Re: Projecting Unicode to ASCII

#14

Post by chx »

Drupal 8 has a transliteration component (=independent of Drupal). It's ... not easy. And that's still a pretty weak implementation. ICU / PHP extension intl is a better one but in general, transliteration is just a bag of hurt.

Re: Projecting Unicode to ASCII

#15

Post by vesinisa »

kuschku wrote: post #9
sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Unidecode may handle Chinese fine, but it definitely handles Western European languages wrong. DIN 5007 Var. 2 specifies that for the purposes of sorting, ö is replaced with oe. This also applies to ä (ae), ü (ue) and ß (ss). This same replacement rule is also used on passports and IDs. Unidecode does not handle this correctly. The unidecode author wrote about this: > In German, there's the typographical convention t…

What do you mean by a Western European language? Germany is usually considered geographically part of Central Europe. Finnish (and Estonian) are of course not Germanic nor even Indo-European languages. But OTOH Swedish, a Germanic language, also uses umlauts and considers them separate letters in collation (Finnish collation rules are actually adopted from Swedish). The most 'correct' mapping of Swedish "Lära sig höra" would be "Lara sig hora". Is Swedish not a Western European language?

Re: Projecting Unicode to ASCII

#18

Post by FabHK »

kuschku wrote: post #9
sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Unidecode may handle Chinese fine, but it definitely handles Western European languages wrong. DIN 5007 Var. 2 specifies that for the purposes of sorting, ö is replaced with oe. This also applies to ä (ae), ü (ue) and ß (ss). This same replacement rule is also used on passports and IDs. Unidecode does not handle this correctly. The unidecode author wrote about this: > In German, there's the typographical convention t…

Unidecode is a projection into the ASCII plane, not your favourite language specific preferred transliteration of certain characters. As such, calling it "definitely [...] wrong" is somewhat overblown, in particular since the author of the library explicitly addresses it.

Re: Projecting Unicode to ASCII

#20

Post by aaaaaaaaaab »

I use this transform chain for asciifying filenames (ridiculous as it is, in 2019 we still can't sync unicode filenames between different OSs):

    uconv -x ':: Any-Latin; :: Latin-ASCII; [:^ASCII:] > \_'
Explanation:

    Any-Latin
Transliterates from non-latin scripts to latin script (e.g. "γραφὴν" --> "graphḕn").

    Latin-ASCII
Tries to asciify characters as much as possible by discarding accents, splitting ligatures, replacing Unicode quotes with regular quotes, © --> (C), etc.

    [:^ASCII:] > \_
Replaces any remaining non-ASCII characters with an underscore.
Post reply on HN