Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

91–100 of 289 posts

Re: UTF-8 Everywhere

#91

Post by bruckie »

DannyB2 wrote: post #78
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Some security-sensitive libraries do this, e.g. https://www.javadoc.io/doc/com.google.common.html.types/type...

Re: UTF-8 Everywhere

#92

Post by masklinn »

snazz wrote: post #6
jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.

> Is java.lang.String still UTF-16?

Yes.

> Is there any plan to fix that?

That's not really possible as strings are defined in terms of char and guarantee O(1) access to UTF16 code units. They might try to switch to "indexed UTF8" (as pypy did in the Python ecosystem whereas "CPython proper" refused to switch to UTF8 with the Python 3 upheaval and went with the death trap that is PEP 393 instead).

Re: UTF-8 Everywhere

#93

Post by shpx »

What I never see mentioned about Unicode is Han Unification

https://en.m.wikipedia.org/wiki/Han_unification

As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk about it.

Re: UTF-8 Everywhere

#94

Post by anderspitman »

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc).

With things like data archival, if I have a hard drive with the library of congress stored in ASCII, I need half a sheet of paper to understand how to decode it.

Whereas apparently UTF8 requires 7k words just to explain why it's important. And that's not even looking at the spec.

Just to be crystal clear, I'm not advocating to not use Unicode, or even use it less. I'm just saying I think it maybe shouldn't count as plain text, since it looks a lot like a relatively complicated binary format to me.

Re: UTF-8 Everywhere

#95

Post by cosarara »

smacktoward wrote: post #17
rakoo wrote: post #14

Maybe it's time for MySQL to make "utf8" actually mean UTF-8 then ( https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-u... )

They probably couldn't even if they wanted to, by this point there will be too much software out there depending on "utf8" meaning "MySQL's weird proprietary hacked-up version of UTF-8". The only real solution is to hammer home the message that "utf8mb4" is what you put into MySQL if you want UTF-8.

There are acual problems too, when switching from utf8mb3 to utf8mb4, because of maximum varchar length in indices: https://stackoverflow.com/questions/48500355/mysql-character...

Re: UTF-8 Everywhere

#96

Post by kazinator »

DagAgren wrote: post #54
heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

You can't even write proper English in ASCII. ASCII is an absolute dead end. It's history. Actually representing human language is HARD. It is also absolutely necessary. Whatever solution you choose is going to be complicated, because it is solving a very complicated problem. Throwing your hands up and going "oh this is too hard, I don't like it" will get you nowhere.

You can't write proper snooty English in ASCII, with diaereses and whatnot.

Re: UTF-8 Everywhere

#97

Post by jeltz »

camgunz wrote: post #76
crazygringo wrote: post #36

Earlier quoted context omitted.

> not really a good reason to support UTF-8 over UTF-16 Of course there is, the fact that if you're dealing only with ASCII characters then it's backwards-compatible. Which is a nice convenience in a great number of situations programmers encounter. The minor details of efficiency of an encoding these days isn't particularly relevant -- sure UTF-16 is better for Chinese, but the average webpage usually does have way…

UTF-8's ASCII compatibility is an anti-feature; it's allowed us to continue to use systems that are encoding naive (in practice ASCII-only). It's no substitute for creating encoding-aware programs, libraries, and systems. The vast majority of text is not in HTML or XML, and there's no reason you can't use Chinese characters in JavaScript besides (your strings and variable/class/component/file names will surely outpac…

The reasons most programmers use English in their source code has nothing to do with file size (for that their are JS minimizes) or supported encodings. It has to do with that two things, English is the most used language in the industry so if you want to cooperate with programmers from other parts of the world English is a good idea and because it frankly looks ugly to mix languages in the same file so when the standard library is in English your source code will be too.

So since most source code is in English (and for JS is minimized) UTF-8 works perfectly there too.

Re: UTF-8 Everywhere

#98

Post by zokier »

JdeBP wrote: post #82
bartwe wrote: post #34

Earlier quoted context omitted.

I assume you mean a limited subset of 7bit ascii ? 33-126

% host -t a $'\015'. 1 \015: 19 bytes, 1+0+0+0 records, response, authoritative, nxdomain query: 1 \015 % It's not as straightforward or sensible as you think. It's case insensitive; it's case preserving; and C0 control characters, SPC, and DEL are allowed. The case differentiating bits for letters are nowadays sometimes used in an attempt to foil attackers. If you want things to look back on and say "I think that X…

I thought DNS allowed any arbitrary byte sequence as label (up to max length limit)

Re: UTF-8 Everywhere

#99

Post by unwind »

mercer wrote: post #35
Svip wrote: post #26

Earlier quoted context omitted.

A better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.

"Ij" is also one sounds represented bij two letters, and I think capitalizing just the 'I' is pretty standard. As a Dutch person myself, I didn't even know that there's a glyph for it! We also have "ei", which sounds the same and was invented to annoy people learning Dutch. Then there's "oe", "eu", "ui". And just to fuck even more with people learning the language, we have "au" and "ou" which also sound the same. Oh,…

Spelling it "dike" helps keep people's minds on the right thing. :)

Re: UTF-8 Everywhere

#100

Post by cryptonector »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

Yes, but the only way to interop multiple scripts on a POSIX filesystem is to use UTF-8. I can forgive people for not realizing that filenames in POSIX are a weird animal: they are NUL-terminated strings of characters (char) in some arbitrary codeset and encoding, but US-ASCII '/' is special.

EDIT: Also, "considered UTF-8 by default almost everywhere" is... not necessarily wrong -- nowadays users should be using UTF-8 locales by default. Maybe "almost everywhere" is an exaggeration, but I wouldn't really know.

Post reply on HN