Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

281–289 of 289 posts

Re: UTF-8 Everywhere

#281

Post by rurban »

marcosdumay wrote: post #103
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Unix paths don’t need to be valid UTF-8 Yet, your shell will treat them like UTF-8 just as well. As will the standard library of almost every programming language, as you noticed. If you open one such file in most text editors, they will render whatever is in it as UTF-8. If you use text manipulating utilities, they will work with it as if it was encoded in UTF-8. It's mostly the Linux kernel that disagrees. Everyt…

Exactly. And they still refuse to acknowledge that treating public names, like a file path, as binary only is a wellknown security issue. Names are identifiers and must be recognizable.

With utf8 it is trivial to create similar looking names and fool the user to think it is a valid name. You know this concept from domain names, using punicode as escape mechanism. But both the kernel and the various libc's are too lazy to treat confusables with escapes, to normalize unicode or to use proper unicode security mechanisms for identifiers. Like mixing scripts, right to left and such.

Eg searching a file path needs to follow unicode rules, as we are dealing with identifiers. I believe my libc, the safeclib, is the only one even offering such functionality.

Likewise the presentation layer on the UI (shell, windows) doesn't present confusables as such, but happily takes i18n seriously. Convenience first, security last.

Apple's previous HFS+ normalized names, the new one is insecure again.

Re: UTF-8 Everywhere

#282

Post by rurban »

oconnor663 wrote: post #163
Spivak wrote: post #140

Earlier quoted context omitted.

I mean it doesn't have to assume an encoding for printing, it just has to have a sane way of turning the path into something human readable. Look you're right that this ship has sailed but ideally we would have decided on a way to display and encode binary for file paths.

I dunno. That sounds like proposing to render "foo.txt" as "Zm9vLnR4dA==" or "[102, 111, 111, 46, 116, 120, 116]" or something. I think you probably meant something like "print the regular characters if the string is UTF-8, or a lossless fallback representation of the bytes otherwise." That's a good idea, and I think a lot of programs do that, but at the same time "if the string is UTF-8" is problematic. There's no r…

No, the proposal is not for crazy encoding schemes, like for domain names, that's up to the presentation layer. The need is to follow the unicode security guidelines for identifiers. A path is an identifier, not binary chunk. Thus it needs to follow some rules. Lately some filesystem drivers agreed, but it's still totally insecure all over.

Re: UTF-8 Everywhere

#283

Post by Grue3 »

gsnedders wrote: post #105
shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

Relatively few people frequently look at different Han languages, and relatively few people are looking at txt files containing Han characters (and I expect those that do are typically running with their OS locale set to one of the Han languages?). Enough CJK HTML content is tagged and heuristics are mostly good enough that incorrect font selection isn't a massive issue on the web, and AFAIK most major word processor…

It's a massive issue on web, I routinely see Japanese posts on Twitter displayed in a Chinese font when they consist only of kanji.

Re: UTF-8 Everywhere

#284

Post by loeg »

masklinn wrote: post #102
ekimekim wrote: post #79

Earlier quoted context omitted.

WTF-8 and WTF-16 are a thing: https://simonsapin.github.io/wtf-8/ Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). Then WTF-8 is what you get if you naively transform invalid UTF-16 into UTF-8. It is a superset of UTF-8. This is very useful when deal…

> Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). If WTF-16 is the ability in potentia to store and return invalid UTF-16 without signalling errors, I don't know that there's any actual UTF-16 system out there to the possible exception of… HFS+ mayb…

APFS continues to normalize codepoints as well.

Re: UTF-8 Everywhere

#285

Post by benibela »

benibela wrote: post #280
robocat wrote: post #203

Earlier quoted context omitted.

AFAIK they just provide type name aliases, which do not enforce or warn of you if you mix the “types”.

They have changed it Now the string types have an encoding and the string themselves, too. When you assign a string to a string variable with a type of a different encoding, the string is automatically converted. But it is causing a huge mess. Especially with existing code. When you have a library using utf-8 and one library using the default codepage, that is not valid anymore. Although you can manually override the…

Here is an example of the mess:

I have a benchmark of various maps in freepascal. The benchmark creates strings of random bytes to use as keys.

A classic key-value store is the sorted TStringList.

Now the benchmark of the TStringList fails. Apparently, because it now assumes the keys are valid utf-8 when using the utf-8 codepage as default codepage.

The default codepage can be changed. When I start the benchmark with LANG=C .. it works with the random byte keys. On Windows, the default codepage is usually latin1, so it would work there, too.

Re: UTF-8 Everywhere

#286

Post by yencabulator »

DannyB2 wrote: post #78
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Go is like that. Not the "mixed within" part, though html/template's AST understands the context where you're using a value and escapes it differently. For example, https://golang.org/pkg/html/template/#HTML

Re: UTF-8 Everywhere

#287

Post by anoncake »

cygx wrote: post #257
anoncake wrote: post #256

Earlier quoted context omitted.

Of course it is. Ligatures aren't characters, they're glyphs that represent multiple characters. Unicode does not encode glyphs, that's simply not its job. No more than encoding what font to use or when to render text in italic.

Which is the whole point of Han unification, the argument being that whether or not a particular line in U+4ECA is horizontal or diagonal is just like that. What's the difference?

To the contrary: What any line in any glyph looks like is of no concern because Unicode doesn't deal with glyphs. It deals with abstract characters that don't have appearances to begin with.

"Α" and "A" look exactly the same (at least in most fonts). But each has its own code point because the GREEK CAPITAL LETTER ALPHA simply isn't the LATIN CAPITAL LETTER A or any other Latin letter.

Re: UTF-8 Everywhere

#288

Post by anoncake »

zajio1am wrote: post #263
anoncake wrote: post #255

Earlier quoted context omitted.

The Latin alphabet is not logographic.

It is not logographic, but characters still have meaning - associtated phonemes. Although this is less clear in English, it is emphasized in other languages. And this mapping is different between languages. So 'c' in English has different meaning to 'c' in Czech.

Not really. Morphemes are considered (defined even) as the smallest unit that has meaning by itself.

Re: UTF-8 Everywhere

#289

Post by ygra »

AnIdiotOnTheNet wrote: post #72
ChrisSD wrote: post #24

Earlier quoted context omitted.

Sure we could declare that but then what? Non-unicode filenames won't suddenly disappear. Operating systems won't suddenly enforce unicode. Filesystems will still allow non-unicode names. Simply declaring it doesn't help anybody. In the meantime your application still needs to handle non-unicode filenames otherwise those malicious ones are free to be malicious.

If unicode had a set of "explictly this byte" codepoints, it should be simple to deal with, just pass the invalid bytes of the filename in that way.

Unicode deals with text, so such a set of codepoints is a non-starter, anyway.
Post reply on HN