Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

Post reply on HN
Print view View original

121–130 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#121

Post by saati »

vbezhenar wrote: post #109
danudey wrote: post #58

Earlier quoted context omitted.

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…

UTF-16 is a simple encoding. It should take a few dozens of LoC to convert to UTF-8. At least if you don’t need extreme performance with AVX, etc.

> UTF-16 is a simple encoding

This isn't true, for starters it's two encodings UTF-16LE and UTF-16BE.

Re: Unicode Normalization Forms: When ö ≠ ö

#122

Post by kingcharles »

cyphar wrote: post #105
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Aside from all of the other issues mentioned, for some languages it's not clear what language something is purely based on codepoints. For languages that have Latin-derived writing systems, it's not uncommon to use English letters (without diacritics) to write the language -- how would that be handled? In addition, thanks to Han unification (though this would've been a problem anyway -- loads of characters would've b…

Yeah, for the same reason that CJK designers often use absolutely abhorrent fonts for English words on packaging and printed media, is the same reason Westerners use terrible fonts for CJK. They are working with a language and culture they have no knowledge of, and think that if the characters look sorta right, then they must be readable and look good.

I've made so many horrible localization errors in the past because I had to translate things into 30 different languages and I can only barely read a handful of them, so I just copy and paste whatever the translators give me.

Incidently, I saw Japanese text recently that was quoting both English AND Arabic in the same sentence. And this was in a block of vertical text. That is literally a worst-case scenario I think. You have RTL-vertical text also containing RTL- and LTR-horizontal text. And unlike English, which when placed into vertical Japanese text, you can essentially choose whether you want the characters going vertically or sideways - you can't do that with Arabic as the letters must be joined together, I don't believe you can break them down and stack them on top of each other.

Why can't we all just convert to Esperanto? ;)

Re: Unicode Normalization Forms: When ö ≠ ö

#123

Post by kingcharles »

chrismorgan wrote: post #68

A fun related issue that could occur: applying NFD to a string can make it longer, so a sanitiser that limits file names to 255 UTF-16 code units but doesn’t first normalise to NFD could fail on HFS+. This could occur on systems that normalise to NFC as well: NFC lengthens some strings, e.g. 𝅗𝅥 (U+1D15E MUSICAL SYMBOL HALF NOTE) normalises to 𝅗𝅥 (U+1D157 MUSICAL SYMBOL VOID NOTEHEAD, U+1D165 MUSICAL SYMBOL COMBININ…

That's a really great point about the string-length and not often addressed. You might even be able to force some sort of buffer overflow with that I guess.

Re: Unicode Normalization Forms: When ö ≠ ö

#125

Post by kingcharles »

heikkilevanto wrote: post #36

Well, if 7-bit US ASCII was good enough for our Lord, it is good enough for me ;-)

Well.. if we're getting technical, the "Old" Testament is written in Hebrew, and the "New" Testament in written in Greek.

The first line of Genesis reads thus: (from right-to-left, although the earliest Hebrew is actually LTR) בְּרֵאשִׁית, בָּרָא אֱלֹהִים, אֵת הַשָּׁמַיִם, וְאֵת הָאָרֶץ.

And the beginning of the Gospel of Mark thus: Ἀρχὴ τοῦ εὐαγγελίου Ἰησοῦ Χριστοῦ υἱοῦ θεοῦ.

(and if we're getting super technical there are a bunch of Aramaic phrases in the Bible that Jesus spoke, although I know little of Aramaic and I don't know how it would have been written in Biblical times between the Greek characters)

So the Lord would be needing those 16-bits after all..

Re: Unicode Normalization Forms: When ö ≠ ö

#126

Post by bjoli »

vbezhenar wrote: post #109
danudey wrote: post #58

Earlier quoted context omitted.

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…

UTF-16 is a simple encoding. It should take a few dozens of LoC to convert to UTF-8. At least if you don’t need extreme performance with AVX, etc.

My experience is exactly like what you say. Until it isn't. UTF16 seems like such a neat idea until it meets reality.

Mostly because of weird interactions of different libraries, languages and operating systems.

Re: Unicode Normalization Forms: When ö ≠ ö

#127

Post by otabdeveloper4 »

gumby wrote: post #120
otabdeveloper4 wrote: post #117

Earlier quoted context omitted.

> It might have been better if the 'code pages' idea was refined instead of eliminated Obviously yes, it would have been better. But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet. So here we are.

> But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet. This is not even remotely true.

Yes it is. The people who made Unicode went out of their way to make life hard and offensive for CJK, Cyrillic, etc., users.

Re: Unicode Normalization Forms: When ö ≠ ö

#128

Post by josefx »

jodrellblank wrote: post #102
DangitBobby wrote: post #85

Earlier quoted context omitted.

It sounds like you're saying that cases should matter in some ways but not in others, which I take no issue with.

Case can have information in it, like color and underline and boldface and italics can carry information. I think it would be clever if Google let me colour my search text and then only found text which was rendered in the same colour, but terrible if colouring my search text was mandatory and it then only found pages with text in the same colour. Likewise terrible if your code editor searched only for code with synt…

> Dog in bold, italics, red, green, uppercase, lowercase, initialcaps, smallcaps, are all the same word.

But that is a feature of the word Dog, try the German words maßen (limited amounts) and massen (large amounts), historically they share the same upper case rendering MASSEN. Now someone versed in German could change to the alternative MASZEN or use the rather modern upper case version of ß. However the default naive (and most of the time correct) conversion between cases looses a significant amount of information.

Re: Unicode Normalization Forms: When ö ≠ ö

#130

Post by divingdragon »

kingcharles wrote: post #122
cyphar wrote: post #105

Earlier quoted context omitted.

Aside from all of the other issues mentioned, for some languages it's not clear what language something is purely based on codepoints. For languages that have Latin-derived writing systems, it's not uncommon to use English letters (without diacritics) to write the language -- how would that be handled? In addition, thanks to Han unification (though this would've been a problem anyway -- loads of characters would've b…

Yeah, for the same reason that CJK designers often use absolutely abhorrent fonts for English words on packaging and printed media, is the same reason Westerners use terrible fonts for CJK. They are working with a language and culture they have no knowledge of, and think that if the characters look sorta right, then they must be readable and look good. I've made so many horrible localization errors in the past becaus…

> Incidently, I saw Japanese text recently that was quoting both English AND Arabic in the same sentence. And this was in a block of vertical text. That is literally a worst-case scenario I think.

Look up Mongolian script and you might change your mind :)

Post reply on HN