Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Projecting Unicode to ASCII

johndcook.com

Post reply on HN
Print view View original

1–10 of 42 posts

Re: Projecting Unicode to ASCII

#2

Post by rurban »

That really is a problem of the search engine. Poincaré should be normalized and stemmed before being indexed and queried. (You don't say projected). Wonder which engine failed to do that.

Re: Projecting Unicode to ASCII

#3

Post by sametmax »

If you limit yourself to the West Europe languages, you can just go with the stdlib:

    >>> import unicodedata
    >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore'))
    eeeaucC
But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic:

    >>> from unidecode import unidecode
    >>> unidecode("北亰")
    'Bei Jing '
Anyway, always remember that str.encode(), str.decode(), open() and many other related callables have an "errors" parameters that allow you to deal with unkown solutions when encoding or decoding:

    >>> print("Père Noël".encode("ascii", errors="ignore"))
    b'Pre Nol'
    >>> print("Père Noël".encode("ascii", errors="replace"))
    b'P?re No?l'
I'll conclude with the mandatory "use Python 3" (3.7 if you can, it has many utf8 fixes: https://vstinner.github.io/posix-locale.html), since you'll be in a world of pain if you deal with non-ascii in Python 2, and EOL is next year :) Tic, Toc...

Re: Projecting Unicode to ASCII

#5

Post by yorwba »

rurban wrote: post #2

That really is a problem of the search engine. Poincaré should be normalized and stemmed before being indexed and queried. (You don't say projected). Wonder which engine failed to do that.

TFA is the author of the "search engine" describing how they learned about having to do normalization.

Re: Projecting Unicode to ASCII

#6

Post by titanix2 »

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Just fixing the French here: it’s Père Noël, not Pére Noël.

Re: Projecting Unicode to ASCII

#7

Post by sametmax »

titanix2 wrote: post #6
sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Just fixing the French here: it’s Père Noël, not Pére Noël.

'Tin la honte.

Re: Projecting Unicode to ASCII

#9

Post by kuschku »

sametmax wrote: post #3

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(),…

Unidecode may handle Chinese fine, but it definitely handles Western European languages wrong.

DIN 5007 Var. 2 specifies that for the purposes of sorting, ö is replaced with oe. This also applies to ä (ae), ü (ue) and ß (ss). This same replacement rule is also used on passports and IDs.

Unidecode does not handle this correctly.

The unidecode author wrote about this:

> In German, there's the typographical convention that an umlaut (the double-dots on: ä ö ü) can be written as an "-e", like with "Schön" becoming "Schoen". But Unidecode doesn't do that-- I have Unidecode simply drop the umlaut accent and give back "Schon".

> (I chose this not because I'm a big meanie, but because generally changing "ü" to "ue" is disastrous for all text that's not in German. Finnish "Hyvää päivää" would turn into "Hyvaeae paeivaeae". And I discourage you from being yet another German who emails me, trying to impel me to consider a typographical nicety of German to be more important than all other languages.)

Re: Projecting Unicode to ASCII

#10

Post by de_Selby »

CapacitorSet wrote: post #8

Did anyone else get the impression that this is a shallow post? I expected some detail about eg. the issues of transliteration, but instead it presented a couple of facts about idempotency in UX.

By John's usual standards it is yes, but it's still interesting IMO
Post reply on HN