Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Character Encoding and UTF-8

blog.fredrb.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Character Encoding and UTF-8

#11

Post by chrismorgan »

lifthrasiir wrote: post #8
jcranmer wrote: post #7

There's... a lot of wrong stuff here. Tackling some of the highlights: > ASCII code pages map the upper 128 positions (0x7F:0xFF) of the ASCII byte. Each page holds a different character set. This is one way internationalisation can be achieved. This is at best a poor explanation, and at worst outright wrong. The actual key thing is charset--there's a wide variety of charsets. Because ASCII is an inherently 7-bit cha…

> Unicode provides a unique code for every character, regardless of the language. Moreover, this is not strictly true even after the generous reinterpretation (assuming "unique-under-normalization", "code point sequence", "abstract character" and "script") because Unicode still doesn't encode some scripts [1]. [1] https://www.unicode.org/standard/unsupported.html

… and Han unification means that you’ll often get one code point representing several different “characters”, and you must convey the language out-of-band (e.g. via an XML or HTML lang attribute) for the text to be correctly understood, sometimes.

https://en.wikipedia.org/wiki/Han_unification

Re: Character Encoding and UTF-8

#12

Post by mcswell »

unixbane wrote: post #3
svnpenn wrote: post #2

> You need to know the encoding of any text, otherwise it’s impossible to decipher the message (although it’s common for applications to assume the encoding). Even with the caveat in parentheses, this is quite misleading. For example, the following line is some text, with no specified encoding: > hello world now, while its true this could be some exotic encoding, or maybe just random binary data, I wouldn't call it i…

The only reason it's easy to decode (as in, by a casual, not requiring information theoretic techniques or something like file(1)), is because almost all popular character encodings have went far out of their way to map the first 128 bytes to ASCII. This idea that text is the common medium / lowest common denominator is a misconception and why UN*X is buggy and half working. Just because it appears easy to read in co…

I have no idea what you mean by "this idea that test is the common medium/ lowest common denominator", nor what these "escaping problems" are (I have to escape ASCII codes 0x00 through 0x1f, I guess, but it's unclear to me why that makes the result unreadable, especially since I hardly ever have to escape anything but \n and maybe \t. And the claim that "UN*X is buggy and half working" is just bizarre.

Re: Character Encoding and UTF-8

#13

Post by boondaburrah »

lifthrasiir wrote: post #8
jcranmer wrote: post #7

There's... a lot of wrong stuff here. Tackling some of the highlights: > ASCII code pages map the upper 128 positions (0x7F:0xFF) of the ASCII byte. Each page holds a different character set. This is one way internationalisation can be achieved. This is at best a poor explanation, and at worst outright wrong. The actual key thing is charset--there's a wide variety of charsets. Because ASCII is an inherently 7-bit cha…

> Unicode provides a unique code for every character, regardless of the language. Moreover, this is not strictly true even after the generous reinterpretation (assuming "unique-under-normalization", "code point sequence", "abstract character" and "script") because Unicode still doesn't encode some scripts [1]. [1] https://www.unicode.org/standard/unsupported.html

Yeah I've got beef with Unicode. It doesn't support CJK*. Since I work in games and there are a lot of Japanese games that want to be sold in the Chinese market (and vice versa), this is A Problem. I don't know where they got off thinking those character sets were the same, because if I treat them the same, I don't get paid.

*in my particular example, you can say unicode doesn't support Japanese, /or/ doesn't support Chinese. The answer depends on what font you're using. "Han Unification" affects more than just those two languages, but that's what I have experience with.

Re: Character Encoding and UTF-8

#14

Post by fredrb »

jcranmer wrote: post #7

There's... a lot of wrong stuff here. Tackling some of the highlights: > ASCII code pages map the upper 128 positions (0x7F:0xFF) of the ASCII byte. Each page holds a different character set. This is one way internationalisation can be achieved. This is at best a poor explanation, and at worst outright wrong. The actual key thing is charset--there's a wide variety of charsets. Because ASCII is an inherently 7-bit cha…

Thank you for these points. I've made some corrections in the post.

> consider that something like à can consist of either a precomposed "à" code point or an "a" + "` diacritic" sequence

If Unicode provides a precomposed combination doesn't it mean that in fact has a code point for every character? Regardless of offering diacritic combination codes?

Re: Character Encoding and UTF-8

#15

Post by cryo »

fredrb wrote: post #14
jcranmer wrote: post #7

There's... a lot of wrong stuff here. Tackling some of the highlights: > ASCII code pages map the upper 128 positions (0x7F:0xFF) of the ASCII byte. Each page holds a different character set. This is one way internationalisation can be achieved. This is at best a poor explanation, and at worst outright wrong. The actual key thing is charset--there's a wide variety of charsets. Because ASCII is an inherently 7-bit cha…

Thank you for these points. I've made some corrections in the post. > consider that something like à can consist of either a precomposed "à" code point or an "a" + "` diacritic" sequence If Unicode provides a precomposed combination doesn't it mean that in fact has a code point for every character? Regardless of offering diacritic combination codes?

From my understanding the precomposed ones don't exist for every character, for latin scripts this might be true but other scripts are more complex.

Simple example is emojies where there isn't a precomposed codepoint for all combinations.

Re: Character Encoding and UTF-8

#16

Post by aorth »

> If someone from Brazil writes a message using the letter é to multiple people, they would read as ة in Arabic, и in Russian and as a corner pipe (╔) if they’re using IBM’s code page 850.

Apologies for the minor nitpick: и is in Cyrillic, not Russian. Cyrillic is the script, Russian is the language. There are other languages that use Cyrillic besides Russian (and the script itself was developed around Greece/Bulgaria before Russia even existed).

For Arabic it's the language and the script so you're OK there!

Re: Character Encoding and UTF-8

#17

Post by unixbane »

mcswell wrote: post #12
unixbane wrote: post #3

Earlier quoted context omitted.

The only reason it's easy to decode (as in, by a casual, not requiring information theoretic techniques or something like file(1)), is because almost all popular character encodings have went far out of their way to map the first 128 bytes to ASCII. This idea that text is the common medium / lowest common denominator is a misconception and why UN*X is buggy and half working. Just because it appears easy to read in co…

I have no idea what you mean by "this idea that test is the common medium/ lowest common denominator", nor what these "escaping problems" are (I have to escape ASCII codes 0x00 through 0x1f, I guess, but it's unclear to me why that makes the result unreadable, especially since I hardly ever have to escape anything but \n and maybe \t. And the claim that "UN*X is buggy and half working" is just bizarre.

The UN*X mantra is that text is the common medium and data should be transferred as plain text, as opposed to any other way of encoding data structures like binary.

Escaping problems as in, you embed data structures into text via JSON or XML, and have to write \uXXXX and \" etc, making it unreadable once again.

No, the claim that UN*X is stable is bizzare.

Re: Character Encoding and UTF-8

#18

Post by unixbane »

svnpenn wrote: post #5
fredrb wrote: post #4

Earlier quoted context omitted.

Right. Impossible might have been an exaggeration, I will fix that. The point is that if you're reading a file with the text "hello world", you can only make out the characters because you know the encoding. Given two completely different encodings that map the same hex values in the message it would be impossible to determine which is the correct string. There is no such thing as plain text.

> Given two completely different encodings that map the same hex values in the message it would be impossible to determine which is the correct string. Sorry, but I don't agree with this either. You can, as a human being (or smart enough AI), look at the result in both encodings, and make an educated guess as to which is correct. If they are wholly different as you say, then one should be gibberish, and one should ma…

What you are feeling is called cognitive dissonance. You have the idea of text so hammered into your mind, that when you realize it's merely a convention that makes it readable in practice without needing to know the encoding, you cannot even concede, despite this being an obvious truth. This phenomenon is called "un*x braindamage".

More or less all possible interpretations of what this person said are correct. But UN*X braindamage also comes with dunning-kruger due to the fact that you've memorized so many factoids after many years and think someone doesn't know what they're talking about when they get them wrong despite their overall idea being correct.

Re: Character Encoding and UTF-8

#19

Post by fredrb »

aorth wrote: post #16

> If someone from Brazil writes a message using the letter é to multiple people, they would read as ة in Arabic, и in Russian and as a corner pipe (╔) if they’re using IBM’s code page 850. Apologies for the minor nitpick: и is in Cyrillic, not Russian. Cyrillic is the script, Russian is the language. There are other languages that use Cyrillic besides Russian (and the script itself was developed around Greece/Bulgari…

Thanks for pointing this out! Should have looked it up before. Fixed in the post.
Post reply on HN