Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

101–110 of 289 posts

Re: UTF-8 Everywhere

#101

Post by pjscott »

anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

Unicode is complicated because the languages it needs to handle are, alas, complicated. UTF-8 is super simple. It's a variable-length encoding for 21-bit unsigned integers. Wikipedia gives a handy table showing how it works:

https://en.wikipedia.org/wiki/UTF-8#Description

Re: UTF-8 Everywhere

#102

Post by masklinn »

ekimekim wrote: post #79
Shebanator wrote: post #66

Earlier quoted context omitted.

WTF-16? I like it...

WTF-8 and WTF-16 are a thing: https://simonsapin.github.io/wtf-8/ Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). Then WTF-8 is what you get if you naively transform invalid UTF-16 into UTF-8. It is a superset of UTF-8. This is very useful when deal…

> Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs).

If WTF-16 is the ability in potentia to store and return invalid UTF-16 without signalling errors, I don't know that there's any actual UTF-16 system out there to the possible exception of… HFS+ maybe?.

Re: UTF-8 Everywhere

#103

Post by marcosdumay »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Unix paths don’t need to be valid UTF-8

Yet, your shell will treat them like UTF-8 just as well. As will the standard library of almost every programming language, as you noticed.

If you open one such file in most text editors, they will render whatever is in it as UTF-8. If you use text manipulating utilities, they will work with it as if it was encoded in UTF-8.

It's mostly the Linux kernel that disagrees. Everything else considers them UTF-8.

Re: UTF-8 Everywhere

#104

Post by jcranmer »

camgunz wrote: post #71
jcranmer wrote: post #40

Earlier quoted context omitted.

> There's not really a good reason to support UTF-8 over UTF-16 Two big reasons: 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examp…

> 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. Eh, realistically if you're doing this, you should be validating it like converting from one encoding to another anyway. I get that people won't and haven't, but that's because UTF-8 has this anti-feature where ASCII is compatible with it, and that's led to a lot of…

> I'll choose whatever encoding I like, thanks.

If everyone chooses whatever encoding they like, then the charset being used has to be encoded somewhere. The problem is, there are lots of places where charset isn't encoded (such as your filesystem). That this is a problem can be missed, because almost all charsets are a strict superset of ASCII (UTF-{7,16} are the only such charsets to be found in the top 99.99% of usage), so it's only when you try your first non-ASCII characters that problems emerge.

Unicode has its share of issues, but at this point, Unicode is the standard for dealing with text, and all i18n-aware code is going to be built on Unicode internally. The only safe way to handle text that has even the remotest change of being i18n-aware is to work with charsets that support all of Unicode, and given its compatibility with ASCII, UTF-8 is the most reasonable one to pick.

If you want to insist on using KOI-8, or ISO-2022-JP, or ISO-8859-1, you're implicitly saying "fuck you" to 2/3 of the world's population since you can't support tasks as basic as "let me write my name" for them.

Re: UTF-8 Everywhere

#105

Post by gsnedders »

shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

Relatively few people frequently look at different Han languages, and relatively few people are looking at txt files containing Han characters (and I expect those that do are typically running with their OS locale set to one of the Han languages?).

Enough CJK HTML content is tagged and heuristics are mostly good enough that incorrect font selection isn't a massive issue on the web, and AFAIK most major word processors include metadata in the file that suffices to distinguish language.

Re: UTF-8 Everywhere

#106

Post by [deleted] »

anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

[deleted]

Re: UTF-8 Everywhere

#107

Post by torstenvl »

DannyB2 wrote: post #78
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Failing that, you could also adopt a naming convention with prefixes to indicate what sort of thing it is you're storing there:

hsCode = hsFromUs(usInputBuffer);

ssStoredCode = ssFromHs(hsCode);

https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...

Re: UTF-8 Everywhere

#108

Post by roelschroeven »

Svip wrote: post #26
dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

A better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.

"ij" is sometimes considered a single letter, but certainly not always. Quoting Wikipedia (https://en.wikipedia.org/wiki/IJ_(digraph)):

"IJ (lowercase ij; Dutch pronunciation: [ɛi]) is a digraph of the letters i and j. Occurring in the Dutch language, it is sometimes considered a ligature, or a letter in itself. In most fonts that have a separate character for ij, the two composing parts are not connected but are separate glyphs, which are sometimes slightly kerned."

(and equivalent in the Dutch Wikipedia article)

Re: UTF-8 Everywhere

#109

Post by nlitened »

anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

As a person who comes from a country with non-ASCII alphabet, I strongly disagree. Since UTF-8 became de-facto standard everywhere, so many headaches went away.

Re: UTF-8 Everywhere

#110

Post by buckminster »

xeeeeeeeeeeenu wrote: post #10
gpvos wrote: post #5

Earlier quoted context omitted.

Have they fixed all the bugs with that pseudocodepage?

Bugs like WriteFile() reporting the wrong number of bytes written with 65001 codepage were fixed years ago.

That's good news. Last time I looked, more than a decade ago admittedly, that bug was WONTFIX.

In fact I was so surprised I just wrote a test program. They have fixed it!

It was the dumbest bug I ever saw in Windows. It was special case code in the console output code path of the user mode part of WriteFile. It only existed to make utf8 work, and it didn't even do that.

Post reply on HN