Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

11–20 of 289 posts

Re: UTF-8 Everywhere

#11

Post by diroussel »

snazz wrote: post #6
jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.

Compact Strings were added in Java 9; https://openjdk.java.net/jeps/254

So they can now be stored as one byte per character.

Re: UTF-8 Everywhere

#12

Post by legulere »

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode

It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either)

Rust is one of the few programming languages that correctly doesn’t treat file paths as strings.

Re: UTF-8 Everywhere

#13

Post by ken »

malkia wrote: post #8

Still doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames. Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.

To me, that's a design flaw. Would we really be any worse off if we simply declared filenames must be UTF-8?

That seems to be the only case where a user-visible and user-editable field is allowed to be an arbitrary byte sequence, and its primary purpose seems to be allowing this argument to pop up on HN every month.

I've never seen any non-malicious use of it. All popular filesystems already disallow specific sets of ASCII characters in names. Any database which needs to save data in files by number has no problem using safe hex filenames.

Re: UTF-8 Everywhere

#15

Post by snazz »

lokedhs wrote: post #9
snazz wrote: post #6

Earlier quoted context omitted.

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.

I think it will be hard to change that. But it's not alone. Javascript also uses UTF-16.

You’re right! I’m surprised I didn’t know that. It looks like it can also be UCS-2, going by the spec:

> A conforming implementation of this International standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it is presumed to be the UTF-16 encoding form.

Re: UTF-8 Everywhere

#16

Post by treve »

rakoo wrote: post #14

Maybe it's time for MySQL to make "utf8" actually mean UTF-8 then ( https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-u... )

> Although utf8 is currently an alias for utf8mb3, at some point utf8 will become a reference to utf8mb4. To avoid ambiguity about the meaning of utf8, consider specifying utf8mb4 explicitly for character set references instead of utf8.

https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8...

Can't fault a database vendor being conservative, but looks like this is planned. Maybe this will be a 9.0 thing.

Re: UTF-8 Everywhere

#17

Post by smacktoward »

rakoo wrote: post #14

Maybe it's time for MySQL to make "utf8" actually mean UTF-8 then ( https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-u... )

They probably couldn't even if they wanted to, by this point there will be too much software out there depending on "utf8" meaning "MySQL's weird proprietary hacked-up version of UTF-8".

The only real solution is to hammer home the message that "utf8mb4" is what you put into MySQL if you want UTF-8.

Re: UTF-8 Everywhere

#18

Post by mormegil »

nathanaldensr wrote: post #3
jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Do you have a source for this? AFAIK the .NET Framework CLR and CoreCLR both still store strings internally as UTF-16.

AFAICT, it's not only "internal representation". .NET strings are defined as a sequence of UTF-16 units, including the definition of the Char type representing a single UTF-16 code unit. I can't imagine how such a change could be implemented (other than changing the internal representation but converting on all accesses which would be nonsense, I think).

Re: UTF-8 Everywhere

#19

Post by rimunroe »

snazz wrote: post #6
jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.

JavaScript:

https://www.ecma-international.org/ecma-262/5.1/#sec-2

> A conforming implementation of this Standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it presumed to be the UTF-16 encoding form.

https://www.ecma-international.org/ecma-262/5.1/#sec-4.3.16

> A String value is a member of the String type. Each integer value in the sequence usually represents a single 16-bit unit of UTF-16 text. However, ECMAScript does not place any restrictions or requirements on the values except that they must be 16-bit unsigned integers.

Re: UTF-8 Everywhere

#20

Post by dpc_pw »

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak.

Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Republic, or is just my definition of "character" wrong.

Post reply on HN