51–60 of 289 posts
> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…
Earlier quoted context omitted.
Are you sure? That will result in a conversion every time a string is passed to the kernel. Windows can handle utf-8 but it is not the native character set for the platform.
There's a conversion in every ...A() function. Conversion between UTF-8 and WTF-16 is just more of the same, but without codepage lookup tables. (-:
Earlier quoted context omitted.
Yeah, but in Czech it's "č".
No. Č is something else, ch is a digraph that's pronounced differently. Take a look at Czech and Slovak alphabets specifically: https://en.wikipedia.org/wiki/Czech_orthography https://en.wikipedia.org/wiki/Slovak_orthography
I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…
Actually representing human language is HARD. It is also absolutely necessary. Whatever solution you choose is going to be complicated, because it is solving a very complicated problem.
Throwing your hands up and going "oh this is too hard, I don't like it" will get you nowhere.
> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…
This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…
> There's not really a good reason to support UTF-8 over UTF-16 Two big reasons: 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examp…
Earlier quoted context omitted.
I think it will be hard to change that. But it's not alone. Javascript also uses UTF-16.
You’re right! I’m surprised I didn’t know that. It looks like it can also be UCS-2, going by the spec: > A conforming implementation of this International standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified…
> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…
That's a fundamental flaw of UNIX.
Earlier quoted context omitted.
To me, that's a design flaw. Would we really be any worse off if we simply declared filenames must be UTF-8? That seems to be the only case where a user-visible and user-editable field is allowed to be an arbitrary byte sequence, and its primary purpose seems to be allowing this argument to pop up on HN every month. I've never seen any non-malicious use of it. All popular filesystems already disallow specific sets of…
Sure we could declare that but then what? Non-unicode filenames won't suddenly disappear. Operating systems won't suddenly enforce unicode. Filesystems will still allow non-unicode names. Simply declaring it doesn't help anybody. In the meantime your application still needs to handle non-unicode filenames otherwise those malicious ones are free to be malicious.