Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

81–90 of 289 posts

Re: UTF-8 Everywhere

#81

Post by masklinn »

dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

> So I wonder if that's different in Slovakia or Chech Republic, or is just my definition of "character" wrong.

According to wikipedia, "Ch" is a character of the Czech alphabet in the sense that it impacts alphabetical ordering ("Ch" sorts between H and I), in the same way Ł or Ę are apparently characters from the Polish alphabet distinct from L and E respectively (wikipedia mentions that "być comes after bycie").

That is unlike, say, french where É and E are the same character alphabetically.

[0] https://en.wikipedia.org/wiki/Czech_orthography

Re: UTF-8 Everywhere

#82

Post by JdeBP »

bartwe wrote: post #34
goatinaboat wrote: post #27

Earlier quoted context omitted.

Certain things such as DNS, email addresses and so on should be restricted to ASCII, it’s a security nightmare otherwise.

I assume you mean a limited subset of 7bit ascii ? 33-126

    % host -t a $'\015'.
    1 \015:
    19 bytes, 1+0+0+0 records, response, authoritative, nxdomain
    query: 1 \015
    %
It's not as straightforward or sensible as you think. It's case insensitive; it's case preserving; and C0 control characters, SPC, and DEL are allowed. The case differentiating bits for letters are nowadays sometimes used in an attempt to foil attackers. If you want things to look back on and say "I think that X was a mistake." then forget UTF of any stripe. The DNS is full of them.

Re: UTF-8 Everywhere

#83

Post by bjourne »

ChrisSD wrote: post #24
ken wrote: post #13

Earlier quoted context omitted.

To me, that's a design flaw. Would we really be any worse off if we simply declared filenames must be UTF-8? That seems to be the only case where a user-visible and user-editable field is allowed to be an arbitrary byte sequence, and its primary purpose seems to be allowing this argument to pop up on HN every month. I've never seen any non-malicious use of it. All popular filesystems already disallow specific sets of…

Sure we could declare that but then what? Non-unicode filenames won't suddenly disappear. Operating systems won't suddenly enforce unicode. Filesystems will still allow non-unicode names. Simply declaring it doesn't help anybody. In the meantime your application still needs to handle non-unicode filenames otherwise those malicious ones are free to be malicious.

As long as the tool for renaming files handles non-utf8 filenames you'd be fine.

Re: UTF-8 Everywhere

#84

Post by masklinn »

Svip wrote: post #26
dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

A better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.

I don't know that that's correct. That there exists a ligature character doesn't mean the ligature is a character of the language.

It could, mind, I don't know dutch. But in french "œ" (which has a ligatured character as you can see) is canonically equivalent to "oe". It is not a separate letter of the alphabet even though:

* many words should not be written with the ligatured form

* many words should be written with the ligatured form

* it has a different pronunciation than the base form

Re: UTF-8 Everywhere

#85

Post by gnarbarian »

DannyB2 wrote: post #78
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

You would probably like Java 1.4

Re: UTF-8 Everywhere

#86

Post by jeltz »

camgunz wrote: post #21

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…

But is it really a plurality? Portuguese, English, Spanish, Turkish, Vietnamese, French, Indonesian and German are stored more efficiently in UTF-8 while Chinese, Korean and Japanese are stored less effeciently. My gut feel is that more people use the Latin script than people using CJK scripts. Indic scripts, Thai, Cyrillic, etc are stored using two bytes in both UTF-8 AND UTF-16.

And thus ignores markup which is in ascii.

Re: UTF-8 Everywhere

#87

Post by jdc »

DannyB2 wrote: post #78
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Cf. newtype in Python and Haskell.

Re: UTF-8 Everywhere

#88

Post by masklinn »

gnarbarian wrote: post #85
DannyB2 wrote: post #78

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

You would probably like Java 1.4

This pattern (newtyping) is a huge weakness of Java in general, and even more so older Java, and people who like newtyping are not going to like java.

Because creating newtypes in Java is

1. verbose, defining a trivial wrapper takes half a dozen lines before you've even done anything

2. slow, because you're paying for the overhead of an extra allocation and pointer indirection every time, unless you jump through unreadable hoops making for even more verbose newtypes[0]

It is a much more convenient (and thus frequent) pattern in languages like Haskell. Or Rust.

[0] https://gist.github.com/jbgi/d6b677d084fafc641fe01f7ffd00591...

Re: UTF-8 Everywhere

#89

Post by jcranmer »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either)

A decent fraction of software can impose rules on the portion of the filesystem within their control. A tool like mv or vim has to be prepared to handle any filepath encoding. But something like a VCS could reasonably insist that they only support filetrees with normalized UTF-8 encoding and no case-insensitive conflicts as the only things reliably working cross-platform.

Re: UTF-8 Everywhere

#90

Post by coolreader18 »

mark-r wrote: post #56
jcranmer wrote: post #40

Earlier quoted context omitted.

> There's not really a good reason to support UTF-8 over UTF-16 Two big reasons: 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examp…

Every program that purports to support Unicode should be tested with a bunch of emoticons.

Do you mean emoji? I don't see what the issue would be with [{}:();P\[\],./~-_+=XD]
Post reply on HN