Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

111–120 of 289 posts

Re: UTF-8 Everywhere

#111

Post by klodolph »

shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding.

However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters.

So for a given “Han character”, there might be between one and three different Unicode characters, and there might be between one and three different ways of writing it.

Here’s an illustration: https://japanese.stackexchange.com/questions/64590/why-are-j...

So the issue does come up when mixing Chinese and Japanese text, but it’s not really one that has a big impact on legibility of the text but you would definitely be concerned if you were writing a Japanese textbook for Chinese students, or vice versa.

Beyond that, it is usually fairly trivial to distinguish between Japanese and Chinese text, so you could just lean on simple heuristics to get the work done (Japanese text, with the exception of fairly ancient text or very short fragments, contains kana, but Chinese does not).

Re: UTF-8 Everywhere

#112

Post by akie »

mercer wrote: post #35
Svip wrote: post #26

Earlier quoted context omitted.

A better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.

"Ij" is also one sounds represented bij two letters, and I think capitalizing just the 'I' is pretty standard. As a Dutch person myself, I didn't even know that there's a glyph for it! We also have "ei", which sounds the same and was invented to annoy people learning Dutch. Then there's "oe", "eu", "ui". And just to fuck even more with people learning the language, we have "au" and "ou" which also sound the same. Oh,…

As a Dutch person myself, capitalizing just the I and not the J hurts my eyes. Ijsselmeer or IJsselmeer?

Re: UTF-8 Everywhere

#113

Post by wtetzner »

qiqitori wrote: post #22
malkia wrote: post #8

Still doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames. Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interoperability

It sounds like they're saying the opposite. All programs dealing with filenames need to be able to support an arbitrary stream of bytes, they can't just assume UTF-8.

Re: UTF-8 Everywhere

#114

Post by oconnor663 »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8

Yes but, most programs expect to be able to print filepaths at least under some circumstances, like printing error messages. Even if a program is fully correct and doesn't assume an encoding in normal operation, it still has to assume one for printing. Filepaths that aren't utf-8 lead to a bunch of ����� in your output (at best). So I think it's fair to say that Unix paths are assumed to be utf-8 by almost all programs, even if being invalid utf-8 doesn't actually cause a correct program to crash.

Re: UTF-8 Everywhere

#115

Post by mhh__ »

DannyB2 wrote: post #78
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Allowed you to? You could do that in C++ quite happily, it's just not useful enough. To bother implementing, at least.

Re: UTF-8 Everywhere

#116

Post by camgunz »

jeltz wrote: post #86
camgunz wrote: post #21

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…

But is it really a plurality? Portuguese, English, Spanish, Turkish, Vietnamese, French, Indonesian and German are stored more efficiently in UTF-8 while Chinese, Korean and Japanese are stored less effeciently. My gut feel is that more people use the Latin script than people using CJK scripts. Indic scripts, Thai, Cyrillic, etc are stored using two bytes in both UTF-8 AND UTF-16. And thus ignores markup which is in…

Looking at the basic multilingual plane [1], UTF-8 will use > 2 bytes to encode essentially anything that isn't:

* ASCII/Latin

* Cyrillic

* Greek

* Most of Arabic

That leaves out:

* China

* India

* Japan

* Korea

* All of Southeast Asia

Re: markup, think about any text that's in a database, stored in RAM, or stored on a disk--relatively little of it will be in noisy ASCII markup formats like HTML or XML.

[1]: https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilin...

Re: UTF-8 Everywhere

#117

Post by masklinn »

qiqitori wrote: post #22
malkia wrote: post #8

Still doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames. Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interoperability

> 2) Bye bye backward compatibility and interoperability

It's already not really a thing.

Traditional unices allow arbitrary bytes with the exception of 00 and 2f, NTFS allows arbitrary utf-16 code units (including unpaired surrogates) with the exception of 0000 and 002f, and I think HFS+ requires valid UTF-16 and allows everything (including NUL).

The OS then adds its own limitations e.g. win32 forbids \, :, *, ", ?, , | (as well as a few special names I think) and OSX forbids 0000 and 003a (":"), the latter of which gets converted to and from "/" (and similarly forbidden) by the POSIX compatibility layer.

The latter is really weird to see in action, if you have access to an OSX machine: open a terminal, try to create a file called "/" and it'll fail. Now create one called ":". Switch over to the Finder, and you'll see that that file is now called "/" (and creating a file called ":" fails).

Oh yeah and ZFS doesn't really care but can require that all paths be valid UTF8 (by setting the utf8only flag).

Re: UTF-8 Everywhere

#118

Post by cryptonector »

shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

First of all, there is no new unification work ongoing. The Unicode Consortium moved on from that by moving on from UCS-2. UCS-2 drove unification as a way to preserve precious codespace.

There used to be language tag codepoints for this, but they've been deprecated. Han unification is an accident of history: a result of UTF-8 not having existed until it was too late!

There's not going to be a different new Unicode for doing away with Han unification, which is why no one mentions it: besides crying about it, what else can one do? Maybe we should revive language tags?

Anyways, isn't the difference between unified Han/Kanji characters mostly stylistic rather than semantic? I'm not denying that many users would get annoyed, but again, what to do about it??

Re: UTF-8 Everywhere

#119

Post by camgunz »

eMSF wrote: post #80
camgunz wrote: post #64

Earlier quoted context omitted.

I'm gonna do little quotes but, I don't mean to be passive aggressive. It's just that this stuff comes up all the time > I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone... This should be the responsibility of a string library internally, and if you're saving data to disk or sending it over the network, you should be ser…

>We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. Mistaking a variable-width encoding for a fixed-width one is specifically a UTF-16 problem. UTF-8 is so obviously not fixed-width that such an error could not happen by a mistake, because even before widespread use of emojis, multibyte sequences were not in any way a corne…

I mean, I think we're both in the realm of [citation needed] here. I would argue that people index into strings quite a lot--whether that's because we thought UCS-2 would be enough for anybody or UTF-8 == ASCII and "it's probably fine" is academic. The solution is the same though: don't index into strings, don't assume an encoding until you've validated. That makes any "advantage" UTF-8 has disappear.

If you really think no one made this mistake with UTF-8, just read up on Python 3.

Re: UTF-8 Everywhere

#120

Post by TheCoelacanth »

anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

You only need one sentence to explain why ASCII isn't sufficient: There are languages other than English.
Post reply on HN