Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

71–80 of 289 posts

Re: UTF-8 Everywhere

#71

Post by camgunz »

jcranmer wrote: post #40
camgunz wrote: post #21

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…

> There's not really a good reason to support UTF-8 over UTF-16 Two big reasons: 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examp…

> 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII.

Eh, realistically if you're doing this, you should be validating it like converting from one encoding to another anyway. I get that people won't and haven't, but that's because UTF-8 has this anti-feature where ASCII is compatible with it, and that's led to a lot of problems.

> 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examples of this.

This is one of those problems, and it's way worse with UTF-8 because it encodes ASCII the same way ASCII does. It's let programmers stay naive about this stuff for... decades?

> Internally, your program should be using UTF-8 (or UTF-16 if you have to for legacy reasons), and you should convert from non-Unicode charsets as soon as possible.

There are all kinds of reasons to not use UTF-8. tialaramex pointed out one above. "UTF-8 everywhere" is simply unrealistic, and it forces a lot of applications to be slower, or to take on unnecessary complexity. Maybe it's worth it to "never have to think about encodings again", but that's pretty hard to verify and there's no way it happens in our lifetimes anyway.

> and you need to have strong justification for why your code needs that complication.

Yeah see, I strongly disagree with this. I'll choose whatever encoding I like, thanks. Maybe you don't mean to be super prescriptive here, but I think a little more consideration by UTF-8 advocates wouldn't hurt.

Re: UTF-8 Everywhere

#72

Post by AnIdiotOnTheNet »

ChrisSD wrote: post #24
ken wrote: post #13

Earlier quoted context omitted.

To me, that's a design flaw. Would we really be any worse off if we simply declared filenames must be UTF-8? That seems to be the only case where a user-visible and user-editable field is allowed to be an arbitrary byte sequence, and its primary purpose seems to be allowing this argument to pop up on HN every month. I've never seen any non-malicious use of it. All popular filesystems already disallow specific sets of…

Sure we could declare that but then what? Non-unicode filenames won't suddenly disappear. Operating systems won't suddenly enforce unicode. Filesystems will still allow non-unicode names. Simply declaring it doesn't help anybody. In the meantime your application still needs to handle non-unicode filenames otherwise those malicious ones are free to be malicious.

If unicode had a set of "explictly this byte" codepoints, it should be simple to deal with, just pass the invalid bytes of the filename in that way.

Re: UTF-8 Everywhere

#73

Post by josefx »

snazz wrote: post #6
jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.

I don't think they can fix that without completely breaking backwards compatibility. The basic char type in Java is defined as a 16 bit wide unsigned integer value and String doesn't abstract over that.

Re: UTF-8 Everywhere

#75

Post by FabHK »

jandrese wrote: post #46
heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

> It is a pain in the ass to have a variable number of bytes per char. Maybe, but nobody can stomach the wasted space you get with UTF-32 in almost every situation. The encoding time tradeoff was considered less objectionable than making most of your text twice or four times larger.

And as the article points out, even then you might have more than one code point for a character.

> For example, the only way to represent the abstract character ю́ cyrillic small letter yu with acute is by the sequence U+044E cyrillic small letter yu followed by U+0301 combining acute accent.

Re: UTF-8 Everywhere

#76

Post by camgunz »

crazygringo wrote: post #36
camgunz wrote: post #21

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…

> not really a good reason to support UTF-8 over UTF-16 Of course there is, the fact that if you're dealing only with ASCII characters then it's backwards-compatible. Which is a nice convenience in a great number of situations programmers encounter. The minor details of efficiency of an encoding these days isn't particularly relevant -- sure UTF-16 is better for Chinese, but the average webpage usually does have way…

UTF-8's ASCII compatibility is an anti-feature; it's allowed us to continue to use systems that are encoding naive (in practice ASCII-only). It's no substitute for creating encoding-aware programs, libraries, and systems.

The vast majority of text is not in HTML or XML, and there's no reason you can't use Chinese characters in JavaScript besides (your strings and variable/class/component/file names will surely outpace your use of keywords).

Re: UTF-8 Everywhere

#77

Post by jeltz »

camgunz wrote: post #64
flohofwoe wrote: post #31

Earlier quoted context omitted.

I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone, and that UTF-16 isn't a fixed-length encoding either). From that perspective, keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, e…

I'm gonna do little quotes but, I don't mean to be passive aggressive. It's just that this stuff comes up all the time > I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone... This should be the responsibility of a string library internally, and if you're saving data to disk or sending it over the network, you should be ser…

> We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation.

So what do you suggest? UTF-16 and UTF-32 encourage this even more.

Re: UTF-8 Everywhere

#78

Post by DannyB2 »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings.

Imagine if languages allowed subtypes of strings which are not directly assignment compatible.

HtmlString

SqlString

String

A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup.

Similarly a String would be converted to a SqlString via a function.

It would be difficult to accidentally mix up strings because they would be assignment incompatible without the functions that translate them.

There could be mixed "languages" within a string. Like a JSP or PHP that might contain scripting snippets, and also JavaScript and CSS snippets, each with different syntax rules and escaping conventions.

Re: UTF-8 Everywhere

#79

Post by ekimekim »

Shebanator wrote: post #66
JdeBP wrote: post #43

Earlier quoted context omitted.

There's a conversion in every ...A() function. Conversion between UTF-8 and WTF-16 is just more of the same, but without codepage lookup tables. (-:

WTF-16? I like it...

WTF-8 and WTF-16 are a thing: https://simonsapin.github.io/wtf-8/

Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs).

Then WTF-8 is what you get if you naively transform invalid UTF-16 into UTF-8. It is a superset of UTF-8.

This is very useful when dealing with applications like Java and Javascript that treat strings as sequences of 16-bit code points, even though not all such strings are valid UTF-16.

Re: UTF-8 Everywhere

#80

Post by eMSF »

camgunz wrote: post #64
flohofwoe wrote: post #31

Earlier quoted context omitted.

I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone, and that UTF-16 isn't a fixed-length encoding either). From that perspective, keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, e…

I'm gonna do little quotes but, I don't mean to be passive aggressive. It's just that this stuff comes up all the time > I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone... This should be the responsibility of a string library internally, and if you're saving data to disk or sending it over the network, you should be ser…

>We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation.

Mistaking a variable-width encoding for a fixed-width one is specifically a UTF-16 problem. UTF-8 is so obviously not fixed-width that such an error could not happen by a mistake, because even before widespread use of emojis, multibyte sequences were not in any way a corner case for UTF-8 text (for additional reference, compare UTF-16 String APIs in Java/JavaScript/etc. with UTF-8 ones in, say, Rust and Go, and see which ones allow you to easily split a string where you shouldn't be able to, or access "half-chars" as a datatype called "char".)

Post reply on HN