Live data from Hacker News

Unicode is harder than you think

mcilloni.ovh

91–100 of 121 posts

Re: Unicode is harder than you think

#91
With the benefit of hindsight, if the standard was done over again either a lot of unnecessary difficulties mentioned in this article should be avoided or the standard split in two. It is arguably too ambitious, too inefficient, and too unrealistic for a number of these things to be handled as recommended in all contexts. There are many examples - in operating system kernels and system programming in general, to name two.

To start with, alternative representations for the same visibly identical character should have been excluded. In the base standard all supported characters should be precomposed - no modifiers. The article points out the difficulties in something as simple (and as security critical) as determining whether two strings refer to the same visibly identical characters.

A character set that does not make that trivial is not suitable for general use in system programming or for security critical identifiers (unfortunately). It massively complicates programming in many programming languages as well, and even UTF-32 is not sufficient to remedy the problem, as the article well notes.

The inability to handle and process any arbitrary string of bytes in a universal character set is a serious problem as well. The world is complicated, and the inability to pass through incorrectly coded data, alternatively coded data, and general binary data is a major limitation with serious consequences. System code such as device drivers or filesystems typically can't deal with inefficiencies or limitations like that.

In addition, there probably should be two types of uppercase and lowercase conversions, a simple, predictable one for system programming, and a more complex one that deals with considerations in languages that do not follow the normal rules.

String collation should be done at two levels as well, a simple code point level suitable for such things as prefix matches on database indexes, and a more complex variation for applications where linguistically sensitive collation is critical.

In general trying to solve all the higher end use cases in a large body of software that do not need to deal with them, should not need to deal with them, or cannot deal with them is impractical and has exacerbated the common string processing issues we see today. A lightweight standard - perhaps a subset or profile of Unicode that supported arbitrary binary data as opaque codepoints - could be helpful in a lot of contexts.

Re: Unicode is harder than you think

#92
post #84

Earlier quoted context omitted.

What is that case? Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how…

> What is that case? ... Hangul. What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)

Thanks!

Re: Unicode is harder than you think

#93
post #56

Earlier quoted context omitted.

The problem is that if the operating system allows sequences that are not valid UTF-16, then you cannot just state "it uses UTF-16": anything that is expecting UTF-16 and only UTF-16 is going to break on valid file names. "That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. I…

Does this actually matter? Who is out there, in the wild, creating file paths that use incredibly cursed random sequences of bytes? Maybe it's just me, but I don't really see the need to accommodate people who do stupid things to see what breaks.

Some unarchivers (especially ones which aren't Unix natives, like say rar) seem to love to do it. They're just buggy, of course.

Beyond that, before the mid-2000s, it was common to use non-UTF-8 locales on Linux. So I'm sure I still have ISO-8859-1/—15 encoded file names somewhere, especially in archived data. They're not always trivial to rename either, because there might be references to them by name. (Or, in odd cases, you can't convert the name to UTF-8 because you hit a filename length limit, since UTF-8 is more bytes).

I believe wanting to access data from 20 years ago is a perfectly reasonable use case.

It's not so bad if a program can't display the file name right, as long as it doesn't crash with an exception or refuse to open the file. Unix file names have been defined as arbitrary sequences of octects except / and NUL for 30+ years.

Re: Unicode is harder than you think

#94

Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "…

Maybe an "Acid test" for Unicode would help? These pages seem to go into that direction: https://www.kermitproject.org/utf8.html https://web.archive.org/web/20160306060703/http://www.inter-... Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.

I was tempted to include unicode when I enhanced the URL/Domain generating strategies in Hypothesis to correctly generate from the set of all valid TLDs a few years ago! But I decided against it since technically the canonical storage form doesn't contain the unicode characters as it should have had Punycode (https://en.wikipedia.org/wiki/Punycode ) conversion done by the "input" system be it a URL bar of your browser or whatever input handling is in play. Assuming all domain processors are able to correctly Punycode was a bit too much, so I kept it to the set of basic valid characters without needing Punycode conversion, and if someone wanted to fuzz test fully they could just add a layer that generates from full unicode.

Re: Unicode is harder than you think

#95
post #84

Earlier quoted context omitted.

What is that case? Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how…

> What is that case? ... Hangul. What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)

I don't understand why the felt the need to add precomposed characters for Hangul. Why? Why couldn't they just let the system compose them instead?

Re: Unicode is harder than you think

#96

Earlier quoted context omitted.

Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…

The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?

Count by grapheme clusters and stop right before it exceeds the limit?

Re: Unicode is harder than you think

#97
post #31
post #29

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "cha…

> "Strings are always UTF-8. Anything else is a bug. Deal with it."

That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.

Re: Unicode is harder than you think

#99

I'll always take an excuse to link to one of my favorite StackOverflow answers, to the question "Why does modern Perl avoid UTF-8 by default?": https://stackoverflow.com/a/6163129/2521092 >. It's from 2011 and Perl-centric, of course, but skip down to "𝔸 𝕤 𝕤 𝕦 𝕞 𝕖 𝔹 𝕣 𝕠 𝕜 𝕖 𝕟 𝕟 𝕖 𝕤 𝕤" for a thorough, if opinionated, list. My favorite, similar to the issue of case-folding, is the idea of something bein…

I loved that answer too but tchrist didn't explain why any of the assumptions were wrong! That led me down a rabbit hole and I did it myself: https://richardjharris.github.io/all-sorts-of-things-you-can...

Re: Unicode is harder than you think

#100
post #3

Favourite unicode fact: properly rendering unicode requires understanding of the current geopolitical situation (Depending on whom you accept as a country and whom you do not, two country-code-letters may or may not render as a flag. This changes sometimes in today's world.). https://esham.io/2014/06/unicode-flags

This is actually a genius decision on their part! It's telling that for Han unification they stomped all over Japanese/Korean/Vietnamese glyphs but still kept the simplified and traditional forms separate.
Post reply on HN