Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is Unicode Designed Badly?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 15 posts

Ask HN: Is Unicode Designed Badly?

#1

Post by indil »

The more I learn about Unicode, the more complicated it gets. It was rather shocking to learn that the presence of combining characters makes most "reverse a string" programming solutions incorrect, and that strings need to be normalized to compare them. The whole thing seems so much more complicated than it should be, but perhaps that's just the nature of the problem?

Was Unicode designed well? If it were designed from scratch today, with no legacy considerations, would the ideal design look like the current design? What would you change?

Being extremely ignorant of the problem space, the first thing I would consider for the chopping block would be combining characters. Just make every character a precomposed character (one code point), so there's no need for normalization. I'm curious if such a scheme could fit every code point into 32 bits, though. Would this be feasible?

Re: Ask HN: Is Unicode Designed Badly?

#3

Post by coldtea »

mardiyah wrote: post #2

"..characters makes most "reverse a string"..." has nothing with Unicode It's thing to do with whether it's Little or Big Endian Unicode as fixed & secured as Ascii at its era

The problem the parent describes (reversing as string) has nothing to do with endianess (that would be trivial to overcome). It's not about the byte order in the architecture, wire protocol, or format -- but about how characters are defined and combined from substituent elements in Unicode itself (way beyond the byte order).

Re: Ask HN: Is Unicode Designed Badly?

#4

Post by Someone »

Combining characters have their issues (https://en.wikipedia.org/wiki/Zalgo_text), but making string reversal trickier isn’t one of them. “Reversing” is an extremely atypical thing to do with text. I think only programming exercises and palindrome searchers do it. Why would your data structure make that easy to do?

For Unicode, a “design from scratch” design would remove duplicate legacy code points. Why have “é” both as a single code point and as ”e” plus a combining character?

It also wouldn’t have any of the deprecated characters (https://en.wikipedia.org/wiki/Unicode_character_property#Dep...)

I also would remove the few special flag code points (https://home.unicode.org/the-past-and-future-of-flag-emoji/)

If “design from scratch” also means “drop the goal of encompassing old character encodings”, more code points probably could go. Why are DOS box characters in Unicode, while Atari/PET, etc, ones aren’t, for example?

Finally, I would look into making it easier to retrieve character class from a code point (the ‘these code points are digits, these are combining marks, etc’ tables are a bit of a wart, and getting rid of them could be useful in small embedded devices).

I doubt a solution exists there that is future proof agains extension of Unicode and doesn’t blow up memory use, though, and am not sure any embedded devices too small to host those tables actually could use that info.

Re: Ask HN: Is Unicode Designed Badly?

#5

Post by HelloNurse »

coldtea wrote: post #3
mardiyah wrote: post #2

"..characters makes most "reverse a string"..." has nothing with Unicode It's thing to do with whether it's Little or Big Endian Unicode as fixed & secured as Ascii at its era

The problem the parent describes (reversing as string) has nothing to do with endianess (that would be trivial to overcome). It's not about the byte order in the architecture, wire protocol, or format -- but about how characters are defined and combined from substituent elements in Unicode itself (way beyond the byte order).

Zalgo text provides a straightforward example of how "reversing a string" is ambiguous and ill-defined: one might conceivably want to transpose "letters" along with their respective piles of combining marks, stacked in the same order, not to move combining marks to adjacent letters in a different order.

And of course what characters are a mark-bearing letter is application-dependent and there might be collateral requirements (e.g. if the reversed text is meant to be displayed swapping closing and opening delimiters such as parentheses: "(note)" -> ")eton(" or "(eton)" ).

Re: Ask HN: Is Unicode Designed Badly?

#6

Post by Charlotte_Buff »

Reversing a string is a useless operation in the real world. Its only application is padding out interview questions. “How to reverse a string” is also an incredibly vague question. What do you actually want me to do? Reverse code points, or code units, or grapheme clusters, or make it look like it’s written backwards? It doesn’t even make sense as a concept in most of the world’s writing systems.

It’s like giving me a list of numbers and asking me to “combine” them. What does that mean? Do I sum them up, or concatenate them, or something else entirely? A lot of string reversal solutions are “incorrect” because there isn’t even a correct question in the first place.

Even with an infinitely large code space, doing away with combining marks and encoding everything as precomposed would be impossible because you cannot have a definitive list of every single combination of letters and diacritics that may mean something to someone. If Unicode had been the first digital character set ever created, it would not contain a single precomposed code point because they are utterly impractical. As such, normalisation – or at least the canonical reordering part of it – is always going to be a necessity.

Re: Ask HN: Is Unicode Designed Badly?

#7

Post by lysergia »

Every computer science problem eventually ends with The Unicode Problem and its various agendas. Personally I avoid Unicode in my editor and use ASCII at all times. If I have to deal with Unicode, I escape it into the relevant ASCII equivalent and normalize things like emojis to ASCII. This avoids various headaches down the line, since Unicode is not cross-compatible across devices and having everything in ASCII is a saner way to approach that.

Re: Ask HN: Is Unicode Designed Badly?

#8

Post by indil »

Someone wrote: post #4

Combining characters have their issues ( https://en.wikipedia.org/wiki/Zalgo_text ), but making string reversal trickier isn’t one of them. “Reversing” is an extremely atypical thing to do with text. I think only programming exercises and palindrome searchers do it. Why would your data structure make that easy to do? For Unicode, a “design from scratch” design would remove duplicate legacy code points. Why have “é” b…

>Why would your data structure make that easy to do?

Reversing a string merely indicates the problem. There are many cases for operating on graphemes instead of code points. For example, deleting the previous grapheme in a text editor when pressing backspace/delete. I think most programmers assume they're dealing with graphemes when they're actually dealing with code points. See, for example, the rune type in Go.

Re: Ask HN: Is Unicode Designed Badly?

#9

Post by acdha »

lysergia wrote: post #7

Every computer science problem eventually ends with The Unicode Problem and its various agendas. Personally I avoid Unicode in my editor and use ASCII at all times. If I have to deal with Unicode, I escape it into the relevant ASCII equivalent and normalize things like emojis to ASCII. This avoids various headaches down the line, since Unicode is not cross-compatible across devices and having everything in ASCII is a…

> This avoids various headaches down the line, since Unicode is not cross-compatible across devices and having everything in ASCII is a saner way to approach that.

Did you mean to say that not all programs support Unicode? It's been a long time — at least a decade — since I ran into a device which doesn't support it at all, as opposed to something like PHP code which has built-in support but didn't enable it.

Re: Ask HN: Is Unicode Designed Badly?

#10

Post by indil »

Charlotte_Buff wrote: post #6

Reversing a string is a useless operation in the real world. Its only application is padding out interview questions. “How to reverse a string” is also an incredibly vague question. What do you actually want me to do? Reverse code points, or code units, or grapheme clusters, or make it look like it’s written backwards? It doesn’t even make sense as a concept in most of the world’s writing systems. It’s like giving me…

>Reversing a string is a useless operation in the real world

I'm not sure why you focused on this one example, which was just meant to indicate the nature of the issue, not cite a broad concrete problem. There are plenty of situations where you'd want to operate on graphemes, not code points, like deleting the previous grapheme in a text editor. It would certainly help programmers write correct code if the two were the same.

>doing away with combining marks and encoding everything as precomposed would be impossible because you cannot have a definitive list of every single combination of letters and diacritics that may mean something to someone

It seems to me it would be trivial to enumerate these combinations, and assign code points to them. For example, the Germanic umlaut is only used with vowels, so that's at most 5 code points.

Post reply on HN