Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

231–240 of 289 posts

Re: UTF-8 Everywhere

#231

Post by gpderetta »

macintux wrote: post #204
cryptonector wrote: post #122

Earlier quoted context omitted.

As I understand it Han unification happened because at the time all there was was UCS-2 -no UTF-16, no UTF-8- so codespace was tight and precious, and that motivated codespace preserving optimizations, of which Han unification is the notable one. To avoid that they needed to have invented UTF-8 many years earlier. Perhaps if the people designing UTF-8 were more diverse they might have felt the necessity to invent UTF…

UTF-8 was sketched on a placemat as a response to a different idea. It seems likely that had it not arisen in a moment of inspiration by a genius, we would be stuck with another inferior design by committee. https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt

Utf-8 is a magnificent hack.

Re: UTF-8 Everywhere

#232

Post by carapace »

tialaramex wrote: post #219
carapace wrote: post #195

Earlier quoted context omitted.

Yeah, this. I have a pat "Unicode Rant" that boils down to this essentially. Having a catalog of standard numbers-to-glyphs (or symbols or whatever, little pictures humans use to communicate with) is awesome and useful (and all ASCII ever was) but trying to digitalize all of human language is much much more challenging.

But human language doesn't stop being "much much more challenging" if you decide not to engage. Sometimes (and this can even be an admirable choice) in some specialist applications it's acceptable to decide you won't embrace the complexity of human language. But in a lot of places where that's fine we already did this with the decimal digits such as in telephone numbers, or UPC/EAN product codes, so we don't need ASC…

But I'm not saying we shouldn't engage, I'm just pointing out that the catalog of lil pictures is the easy part of the task.

One way I put it is, imagine if one of the first-class outputs of the Unicode Consortium was standard libraries for different human languages for different computer languages.

Re: UTF-8 Everywhere

#233

Post by skissane »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid

How about a new mount option utf8_only? When that is set on a volume, the VFS would block any attempt to create a new file/directory if the name isn't valid UTF-8. (Pre-existing file/directories with invalid UTF-8 can still be accessed.) Distributions could set it by default on all filesystems, but a user could turn it off if it caused a problem for them (which in practice is probably going to be rare.)

One could also have a flag set on the filesystem (e.g. in the superblock) similar to utf8_only. It could only be set at filesystem creation time. If it is set, then any invalid UTF-8 in a filename is a filesystem corruption which fsck could repair. A filesystem with such a flag set would ban invalid UTF-8 irrespective of any utf8_only mount option.

If we are going to ban invalid UTF-8, it would be a good idea for security reasons to ban C0 controls as well (i.e. all characters in range U+0001 to U+001F), see [1]. This could be included in the utf8_only mount option / filesystem flag, or be an independent mount option / filesystem flag. If going with the same flag for both, maybe "sane_filenames_only" might be a better name.

(Actually, for security, one should ban the UTF-8 encodings of the C1 controls as well... the CSI character U+009B might be interpreted as an ESC[ by some applications, which could have nefarious consequences. Likewise, the APC (application program command) and OSC (operating system command) characters could cause security issues, although in practice support for them is rather limited, which limits the scope of the security issues they pose.)

[1] https://www.austingroupbugs.net/view.php?id=251

Re: UTF-8 Everywhere

#234

Post by [deleted] »

mmm_grayons wrote: post #229
Symbiote wrote: post #218

Earlier quoted context omitted.

English isn't even ASCII anyway. Some loanwords like façade or café retain their accents. Units like ° £ € and symbols like © ® × ÷ ½ aren't ASCII. It doesn't take much to need one of these cases in a project.

I know almost no one who actually types the accented e, let alone the c with the cedilla. I scarcely ever see the degree symbol typed. Rather, I see facade, cafe, and "degrees". That aside, the big problem with unicode is not those characters; they're a simple two-byte extension. They obey the simple bijective mapping of binary character character on screen. Unicode doesn't. You have to deal with multiple code points…

[deleted]

Re: UTF-8 Everywhere

#235

Post by cryptonector »

macintux wrote: post #204
cryptonector wrote: post #122

Earlier quoted context omitted.

As I understand it Han unification happened because at the time all there was was UCS-2 -no UTF-16, no UTF-8- so codespace was tight and precious, and that motivated codespace preserving optimizations, of which Han unification is the notable one. To avoid that they needed to have invented UTF-8 many years earlier. Perhaps if the people designing UTF-8 were more diverse they might have felt the necessity to invent UTF…

UTF-8 was sketched on a placemat as a response to a different idea. It seems likely that had it not arisen in a moment of inspiration by a genius, we would be stuck with another inferior design by committee. https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt

I agree. But too, necessity is the mother of invention. GP seems to argue that Han unification happened because the UC was not diverse enough. Maybe, and maybe if it had been diverse enough the need would have arisen sooner. But again, the thing they came up with could have been garbage, who knows!

What I do know is that UTF-8 is genius. The Han unification problems seem mostly minor -- I suspect code can detect language and do the right thing, for example, and again, we could revive language tags if need be.

Re: UTF-8 Everywhere

#236

Post by cryptonector »

wheybags wrote: post #147
cryptonector wrote: post #118

Earlier quoted context omitted.

First of all, there is no new unification work ongoing. The Unicode Consortium moved on from that by moving on from UCS-2. UCS-2 drove unification as a way to preserve precious codespace. There used to be language tag codepoints for this, but they've been deprecated. Han unification is an accident of history: a result of UTF-8 not having existed until it was too late! There's not going to be a different new Unicode f…

It's different enough that users will immediately complain if you get it wrong. And it means that you, as a developer who might not understand either Chinese or Japanese, now has to deal with the fallout by setting a different font in your application depending on which of the two languages it is. This happe ed end for us in factorio, and it was super annoying, because it's really hard to spot the problem before it g…

See the whole point about history. All you're doing is crying about it :(

Here's a question: when a native Chinese speaker reads a Japanese text, do they want to see it in Chinese style or Japanese style? If the former, then just know that that's their preference and always use their preference -- easy fix. If the latter... you need to know the language of a text (or sub-text), and that requires either language tags or language recognition.

I expect it's the latter, to make it easier to recognize foreign text, which is not necessarily easy to read. After all, native Chinese, Japanese, and Korean speakers who don't speak the other languages can only glean so much meaning from Han/Kanji text in the others' languages. That's because while often ideographic characters are used for (common) meaning, sometimes they are used for the sounds of the words they identify but not their meanings.

Re: UTF-8 Everywhere

#237

Post by thanksforfish »

> Many third-party libraries for Windows do not support Unicode: they accept narrow string parameters and pass them to the ANSI API. Sometimes, even for file names. In the general case, it is impossible to work around this, as a string may not be representable completely in any ANSI code page (if it contains characters from a mix of Unicode blocks). What is normally done by Windows programmers for file names is getting an 8.3 path to the file (if it already exists) and feeding it into such a library. It is not possible if the library is supposed to create a non-existing file.

Yikes. That's a fascinating use of 8.3 paths. Sometimes when I look at really old Windows cruft I wonder when it will go away. 8.3 paths seemed like an easy thing to get rid of, but with 8.3 paths used to hack around encoding issues in 3rd party libraries... that's going to stick around...

Anyone know which libraries this is talking about?

Re: UTF-8 Everywhere

#238

Post by cryptonector »

microtherion wrote: post #189
ksec wrote: post #171

Earlier quoted context omitted.

The same could be said whether è é should be the same as e with different fonts. People who cares about it would complain. To those who only uses English it is only the same e .

I don't think that's the same, because e.g. in French, e, é, è, and ê are all used, with different pronunciations.

Not just different pronunciations. ê isn't about pronunciation but about indicating that "there used to be an 's' after this e". French written w/o circumflex accents doesn't change in pronunciation, not really, nor -mostly- in meaning, but it does look very annoying to French speakers as well as to native speakers of other Romance languages: the reason is that that reminder is in fact useful for translation.

I'm guessing Han unification is at least annoying like losing circumflex accents would be.

Re: UTF-8 Everywhere

#239

Post by cryptonector »

innocenat wrote: post #131
cryptonector wrote: post #118

Earlier quoted context omitted.

First of all, there is no new unification work ongoing. The Unicode Consortium moved on from that by moving on from UCS-2. UCS-2 drove unification as a way to preserve precious codespace. There used to be language tag codepoints for this, but they've been deprecated. Han unification is an accident of history: a result of UTF-8 not having existed until it was too late! There's not going to be a different new Unicode f…

It's only stylistic issue if you also consider a and α (alpha) to also be just stylistic different. I have learned to live with it, but it is very annoying.

Do you want to see all Han texts in your preferred style, or do you want to see them in the style corresponding to the language they were written in?
Post reply on HN