Live data from Hacker News

Projecting Unicode to ASCII

johndcook.com

41–42 of 42 posts

Re: Projecting Unicode to ASCII

#41
Romanising Chinese is not as easy as unidecode would made it seem! "銀行" is pronounced "yin hang", but the second character "行" is "xing" when it's alone. This problem is made worse by the lack of spaces when writing Chinese.

Pingtype tries to solve all these problems. If there's a need for it to be ported to Python/etc then I'd be happy to do so!

https://pingtype.github.io

Re: Projecting Unicode to ASCII

#42
post #24

Earlier quoted context omitted.

Why not transform them into utf-8, say with iconv?

How would that help when e.g. copying files from a Linux server to macOS via Samba? Mind you, the filenames can contain both NFC, NFD or the mixture of the two.

Well you're talking about filenames, right? Not the content?

My understanding (which could be quite wrong) is that Windows refers NFC and MacOS prefers NFD (as do I but I understand the NFC desire) but that the filesystems themselves do not do normalization. In which case, modulo delimiters, every filename in one is legit in the other.

My suggestion of utf8 was to make a byte-order and code point width invariant form that should be completely reversible.

But if that doesn't work, never mind.

Post reply on HN