UTF-8 Everywhere

Post by innocenat » Tue, Apr 14, 2020, 7:22 PM UTC

cryptonector wrote: post #118

shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

First of all, there is no new unification work ongoing. The Unicode Consortium moved on from that by moving on from UCS-2. UCS-2 drove unification as a way to preserve precious codespace. There used to be language tag codepoints for this, but they've been deprecated. Han unification is an accident of history: a result of UTF-8 not having existed until it was too late! There's not going to be a different new Unicode f…

Post by mark-r » Tue, Apr 14, 2020, 7:28 PM UTC

camgunz wrote: post #119

eMSF wrote: post #80

Earlier quoted context omitted.

>We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. Mistaking a variable-width encoding for a fixed-width one is specifically a UTF-16 problem. UTF-8 is so obviously not fixed-width that such an error could not happen by a mistake, because even before widespread use of emojis, multibyte sequences were not in any way a corne…

I mean, I think we're both in the realm of [citation needed] here. I would argue that people index into strings quite a lot--whether that's because we thought UCS-2 would be enough for anybody or UTF-8 == ASCII and "it's probably fine" is academic. The solution is the same though: don't index into strings, don't assume an encoding until you've validated. That makes any "advantage" UTF-8 has disappear. If you really t…

Post by arendtio » Tue, Apr 14, 2020, 7:28 PM UTC

marcosdumay wrote: post #103

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Unix paths don’t need to be valid UTF-8 Yet, your shell will treat them like UTF-8 just as well. As will the standard library of almost every programming language, as you noticed. If you open one such file in most text editors, they will render whatever is in it as UTF-8. If you use text manipulating utilities, they will work with it as if it was encoded in UTF-8. It's mostly the Linux kernel that disagrees. Everyt…

Post by Spivak » Tue, Apr 14, 2020, 7:31 PM UTC

marcosdumay wrote: post #103

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Unix paths don’t need to be valid UTF-8 Yet, your shell will treat them like UTF-8 just as well. As will the standard library of almost every programming language, as you noticed. If you open one such file in most text editors, they will render whatever is in it as UTF-8. If you use text manipulating utilities, they will work with it as if it was encoded in UTF-8. It's mostly the Linux kernel that disagrees. Everyt…

Post by msla » Tue, Apr 14, 2020, 7:35 PM UTC

TheCoelacanth wrote: post #120

anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

You only need one sentence to explain why ASCII isn't sufficient: There are languages other than English.