Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

261–270 of 289 posts

Re: UTF-8 Everywhere

#261

Post by perilunar »

carapace wrote: post #195
pjscott wrote: post #101

Earlier quoted context omitted.

Unicode is complicated because the languages it needs to handle are, alas, complicated. UTF-8 is super simple. It's a variable-length encoding for 21-bit unsigned integers. Wikipedia gives a handy table showing how it works: https://en.wikipedia.org/wiki/UTF-8#Description

Yeah, this. I have a pat "Unicode Rant" that boils down to this essentially. Having a catalog of standard numbers-to-glyphs (or symbols or whatever, little pictures humans use to communicate with) is awesome and useful (and all ASCII ever was) but trying to digitalize all of human language is much much more challenging.

> and all ASCII ever was

Except that's not true. The ASCII control codes were never glyphs, but were used to control the hardware.

Re: UTF-8 Everywhere

#262

Post by zajio1am »

oconnor663 wrote: post #163
Spivak wrote: post #140

Earlier quoted context omitted.

I mean it doesn't have to assume an encoding for printing, it just has to have a sane way of turning the path into something human readable. Look you're right that this ship has sailed but ideally we would have decided on a way to display and encode binary for file paths.

I dunno. That sounds like proposing to render "foo.txt" as "Zm9vLnR4dA==" or "[102, 111, 111, 46, 116, 120, 116]" or something. I think you probably meant something like "print the regular characters if the string is UTF-8, or a lossless fallback representation of the bytes otherwise." That's a good idea, and I think a lot of programs do that, but at the same time "if the string is UTF-8" is problematic. There's no r…

> 2) There is no reliable way to determine the encoding of a string, just by looking at its bytes. And Unix doesn't provide any other metadata. 3) Therefore, useful Unix programs must assume that any path that could be UTF-8, is UTF-8, for the purpose of displaying it to the user.

No, there is locale settings (in envvars) and software should assume path encoding based on locale encoding.

It is true that today locale setting is usualy utf-8 based, but if i use non-utf-8 based locale then tools should not assume paths are in utf-8 and recode in.

Re: UTF-8 Everywhere

#263

Post by zajio1am »

anoncake wrote: post #255
tialaramex wrote: post #248

Earlier quoted context omitted.

So what you're saying is that because 'chat' in English and 'chat' in French are quite different words with very different meanings, you believe there should be a separate letter 'c' for English and French to enable us to tell those words apart?

The Latin alphabet is not logographic.

It is not logographic, but characters still have meaning - associtated phonemes. Although this is less clear in English, it is emphasized in other languages.

And this mapping is different between languages. So 'c' in English has different meaning to 'c' in Czech.

Re: UTF-8 Everywhere

#264

Post by zajio1am »

dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

A a native Czech speaker, i never really understood what it means that 'ch' is one letter in Czech. It is clearly two graphemes representing one phoneme, so one could think it is a digraph, but it has some special properties like being one element in collating order. I think people just started to call it one letter to have one-letter-one-sound property.

Re: UTF-8 Everywhere

#265

Post by account42 »

TheCoelacanth wrote: post #198
ftvy wrote: post #151

Earlier quoted context omitted.

When I wrote a very primitive UTF-8 library, I really began to appreciate UTF-8's design. For example; the first byte says how many bytes the character requires. At first it was daunting, but when I put 2 and 2 together, it really opened up. I am sure there are many aspects I am missing about UTF-8, but it is all reasonable in its design and implementation. For reference, I was converting between code points and actu…

The self-synchronizing property is also very clever. If you start at an arbitrary byte, you can find the start of the next character by scanning forward a maximum of 3 bytes.

And scanning backwards works too.

Re: UTF-8 Everywhere

#266

Post by imtringued »

acdha wrote: post #200
jcranmer wrote: post #89

Earlier quoted context omitted.

> It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) A decent fraction of software can impose rules on the portion of the filesystem within their control. A tool like mv or vim has to be prepared to handle any filepath encoding. But something like a VCS…

The history of Git and Subversion handling filenames makes me think that the opposite is true: A VCS which doesn't handle arbitrary byte-strings will have weird edge cases which prevent users from adding files or accessing them, possibly even “losing” data in a local checkout. This is especially tedious because it'll appear to work for a while until someone first tries to commit an unusual file or checks it out with…

The actual text isn't an arbitrary byte string. There is logical data and then there is its representation. char, short, int, string can all logically refer to the number 0 but the representation is completely different. With char it is even possible to represent the same number in two ways. As a binary 0 or as the character code for 0. Allowing byte strings as the physical representation is not a bad idea to stay future proof but you will have to provide additional information by storing the character encoding that was used to create the arbitrary byte string. If you fail to do that then this information will have to provided through convention and that's how we get "stuck" with UTF-8 and I although I like UTF-8 this doesn't feel like the right solution. If everyone agrees to use UTF-8 then we should stop pretending that something is just an arbitrary byte string and formalize UTF-8.

The idea of an arbitrary byte string is fooling people into believing something that is not true. Developers falsely think their software can handle any character encoding. However, once you decide to support only a single character encoding you will notice that if something better comes along you need a way to differentiate the old and new codec. Then you decide to add a field that declares the character encoding type and suddenly it's obvious that your arbitrary byte string is a bad way of dealing with the problem. That byte string has meaning. Don't throw that meaning away.

Re: UTF-8 Everywhere

#267

Post by imtringued »

naniwaduni wrote: post #226
msla wrote: post #58

Earlier quoted context omitted.

It's a reflection of the fact people aren't going to throw out existing filesystems because they aren't in a specific character encoding. There's nothing the OS can do about that, there's nothing programmers in general can do about that, and the only way to fix it is with a time machine and enough persuasion to force everyone to implement Unicode and UTF-8 to the exclusion of any other character encoding schemes.

And it would still be wrong, because the rules of what constitutes valid unicode have changed (what's a surrogate?), and also why would that be a good idea to bake into your filesystem??

It would be a very good idea to acknowledge the existence of codecs by storing the identifier of the chosen codec but forcing a specific one doesn't appear to be that useful.

Re: UTF-8 Everywhere

#268

Post by imtringued »

droopyEyelids wrote: post #141
jaseemabid wrote: post #138

Earlier quoted context omitted.

ASCII is English and limiting access to knowledge for the rest of humanity for a simpler encoding is just not an acceptable option. Someone needs to interpret those 7k words and write a (complicated?) program once so that billions can read in their own language? Sounds like an easy win to me.

counterpoint: A complicated program is never an easy win, and English is already spoken in every country in the world.

You didn't say spoken by every person. Merely spoken in every country. Even the existence of tourists in a country would pass this incredibly low bar...

Re: UTF-8 Everywhere

#269

Post by imtringued »

tachyonbeam wrote: post #182
crazygringo wrote: post #156

Earlier quoted context omitted.

So your argument is... it's easier to teach billions of people fluent English... than for software to support UTF-8? You are aware that a majority of the world's population speaks no English whatsoever?

Playing the devil's advocate here. I am not a native English speaker, I'm a French speaker, but I'm happy that English is kind of the default international language. It's a relatively simple language. I actually make less grammar mistakes in English than I do in my native language. I suppose it's probably not a politically correct thing to say, the English are the colonists, the invaders, the oppressors, but eh, mayb…

For some reason people seem to think that the only options are UTF-8 and ASCII. That choice never existed. There are thousands upon thousands of character encodings in use. Before Unicode every single writing system had its own character encoding that is incompatible with everything else.

Re: UTF-8 Everywhere

#270

Post by account42 »

masklinn wrote: post #117
qiqitori wrote: post #22

Earlier quoted context omitted.

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interoperability

> 2) Bye bye backward compatibility and interoperability It's already not really a thing. Traditional unices allow arbitrary bytes with the exception of 00 and 2f, NTFS allows arbitrary utf-16 code units (including unpaired surrogates) with the exception of 0000 and 002f, and I think HFS+ requires valid UTF-16 and allows everything (including NUL). The OS then adds its own limitations e.g. win32 forbids \, :, *, ", ?…

> Traditional unices allow arbitrary bytes with the exception of 00 and 2f, NTFS allows arbitrary utf-16 code units (including unpaired surrogates) with the exception of 0000 and 002f.

For just Windows -> Linux you can represent everything by mapping WTF-16 to WTF-8.

Post reply on HN