Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Projecting Unicode to ASCII

johndcook.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Projecting Unicode to ASCII

#31

Post by hjek »

akie wrote: post #28
mci wrote: post #13

Earlier quoted context omitted.

Fredrik Lundh describes a DIY patch on top of unicodedata here: http://effbot.org/zone/unicode-convert.htm It solves the ä->ae problem and its ilk for Western European languages. Time has devoured my comment on that post, which extended the transliteration table to Eastern European languages and proposed to use mnemonic names like int(u'\N{Latin capital letter AE}') instead of 0xc6.

ä -> ae is a German transliteration that would not be recognized by a non-German (e.g. a Dutchman).

While a similar transliteration (ø-> oe) would be understood by Danes, it can create misunderstandings: køn (pretty) != koen (the cow), søn (son) != soen (the sow), røde (red) != roede (rowed), tør (dry) != toer (a two) and rør (pipe) != roer (rower).

Re: Projecting Unicode to ASCII

#32

Post by aaaaaaaaaab »

gumby wrote: post #24
aaaaaaaaaab wrote: post #20

I use this transform chain for asciifying filenames (ridiculous as it is, in 2019 we still can't sync unicode filenames between different OSs): uconv -x ':: Any-Latin; :: Latin-ASCII; [:^ASCII:] > \_' Explanation: Any-Latin Transliterates from non-latin scripts to latin script (e.g. "γραφὴν" --> "graphḕn"). Latin-ASCII Tries to asciify characters as much as possible by discarding accents, splitting ligatures, replaci…

Why not transform them into utf-8, say with iconv?

How would that help when e.g. copying files from a Linux server to macOS via Samba? Mind you, the filenames can contain both NFC, NFD or the mixture of the two.

Re: Projecting Unicode to ASCII

#33

Post by samf »

I use pastebot on the mac. It should be easy to add this as a "shell script filter", but sandboxing prevents me from getting unidecode. Has anyone else accomplished this?

Re: Projecting Unicode to ASCII

#34

Post by jwilk »

aaaaaaaaaab wrote: post #20

I use this transform chain for asciifying filenames (ridiculous as it is, in 2019 we still can't sync unicode filenames between different OSs): uconv -x ':: Any-Latin; :: Latin-ASCII; [:^ASCII:] > \_' Explanation: Any-Latin Transliterates from non-latin scripts to latin script (e.g. "γραφὴν" --> "graphḕn"). Latin-ASCII Tries to asciify characters as much as possible by discarding accents, splitting ligatures, replaci…

What does "::" mean?

Re: Projecting Unicode to ASCII

#35

Post by pudo »

I've got to recommend PyICU here. unidecode is good, but has some holes (I noticed the Azeri letter schwa, for example). PyICU is a binding for IBMs International Components for Unicode, and basically has a coding language for unicode transforms. Here's an example that is equivalent to unidecode:

https://github.com/pudo/normality/blob/master/normality/tran...

Re: Projecting Unicode to ASCII

#37

Post by rurban »

yorwba wrote: post #5
rurban wrote: post #2

That really is a problem of the search engine. Poincaré should be normalized and stemmed before being indexed and queried. (You don't say projected). Wonder which engine failed to do that.

TFA is the author of the "search engine" describing how they learned about having to do normalization.

I see, thanks. Now I know which search to avoid. This unicode business should be the most trivial problem in building a search engine. What will he with East-Asian languages? Transliterate to English ASCII? Pretty sure he will skip them. At least there's no stemming there.

Re: Projecting Unicode to ASCII

#38

Post by jwilk »

gumby wrote: post #26

iconv is the standard utility for this (a standard unix/linux utility) and includes the projection of characters that aren't in the target character set. Not sure how well it handles CJKV chars though.

It doesn't do anything useful with them:

  $ echo 北亰 | iconv -t ASCII//TRANSLIT
  ??

Re: Projecting Unicode to ASCII

#39

Post by aaaaaaaaaab »

jwilk wrote: post #34
aaaaaaaaaab wrote: post #20

I use this transform chain for asciifying filenames (ridiculous as it is, in 2019 we still can't sync unicode filenames between different OSs): uconv -x ':: Any-Latin; :: Latin-ASCII; [:^ASCII:] > \_' Explanation: Any-Latin Transliterates from non-latin scripts to latin script (e.g. "γραφὴν" --> "graphḕn"). Latin-ASCII Tries to asciify characters as much as possible by discarding accents, splitting ligatures, replaci…

What does "::" mean?

It’s part of uconv’s transform rule syntax:

>Each transform rule consists of two colons followed by a transform name.

http://userguide.icu-project.org/transforms/general#TOC-Comp...

Re: Projecting Unicode to ASCII

#40

Post by Aloha »

hjek wrote: post #31
akie wrote: post #28

Earlier quoted context omitted.

ä -> ae is a German transliteration that would not be recognized by a non-German (e.g. a Dutchman).

While a similar transliteration (ø-> oe) would be understood by Danes, it can create misunderstandings: køn (pretty) != koen (the cow), søn (son) != soen (the sow), røde (red) != roede (rowed), tør (dry) != toer (a two) and rør (pipe) != roer (rower).

Context though is the missing piece to all of those, all languages, particularly english however is context driven.
Post reply on HN