Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

21–30 of 289 posts

Re: UTF-8 Everywhere

#21

Post by camgunz »

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively.

But more broadly, being able to assume $encoding everywhere is unrealistic. Write your programs/whatevers allowing your users to be aware of and configure encodings. It might not be ideal, but such is life.

Re: UTF-8 Everywhere

#22

Post by qiqitori »

malkia wrote: post #8

Still doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames. Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames?

1) Why?

2) Bye bye backward compatibility and interoperability

Re: UTF-8 Everywhere

#24

Post by ChrisSD »

ken wrote: post #13
malkia wrote: post #8

Still doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames. Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.

To me, that's a design flaw. Would we really be any worse off if we simply declared filenames must be UTF-8? That seems to be the only case where a user-visible and user-editable field is allowed to be an arbitrary byte sequence, and its primary purpose seems to be allowing this argument to pop up on HN every month. I've never seen any non-malicious use of it. All popular filesystems already disallow specific sets of…

Sure we could declare that but then what? Non-unicode filenames won't suddenly disappear. Operating systems won't suddenly enforce unicode. Filesystems will still allow non-unicode names.

Simply declaring it doesn't help anybody. In the meantime your application still needs to handle non-unicode filenames otherwise those malicious ones are free to be malicious.

Re: UTF-8 Everywhere

#25

Post by heyplanet »

I think UTF-8 was a mistake.

It is a pain in the ass to have a variable number of bytes per char.

In Ascii, you could easily know every character personally. No strange surprises.

Also no surprises while reading black on white text and suddenly being confronted with clors [1].

[1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in colors, but it was stripped out. Looks like the makers of HN don't like UTF-8 either.

Re: UTF-8 Everywhere

#26

Post by Svip »

dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

A better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.

Re: UTF-8 Everywhere

#27

Post by goatinaboat »

heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

Certain things such as DNS, email addresses and so on should be restricted to ASCII, it’s a security nightmare otherwise.

Re: UTF-8 Everywhere

#28

Post by ken »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> one of the few programming languages that correctly doesn’t treat file paths as strings

I hear: one of those few programming languages that, despite its vaunted type-safety, makes it possible to accidentally create a file with a completely bogus name that I won't be able to view or open correctly with half the programs on my computer.

Languages which allow arbitrary byte sequences in paths are the cause of, and solution to, all of Unix's pathname problems.

Re: UTF-8 Everywhere

#29

Post by thechao »

heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

You're conflating code points and some encoding; more importantly, you're conflating "array of encoded objects (bytes)" for "a string of text". They're not — and never have been — the same.

Re: UTF-8 Everywhere

#30

Post by lilyball »

ken wrote: post #28
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> one of the few programming languages that correctly doesn’t treat file paths as strings I hear: one of those few programming languages that, despite its vaunted type-safety, makes it possible to accidentally create a file with a completely bogus name that I won't be able to view or open correctly with half the programs on my computer. Languages which allow arbitrary byte sequences in paths are the cause of, and sol…

So what you're saying is the language should not be able to work with pre-existing files whose names are not valid UTF-8?
Post reply on HN