Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

Post reply on HN
Print view View original

121–130 of 159 posts

Re: Unicode Is Awesome

#121

Post by edflsafoiewq »

jcranmer wrote: post #59
TheDong wrote: post #33

Earlier quoted context omitted.

There's a not-insignificant number of Japanese websites that can only correctly display using EUC-JP or ShiftJIS. It seems very Latin/ASCII centric to push for disabling non-UTF-8 encodings, especially since the only reason UTF-8 works so well on ASCII websites is due to its backwards compatibility. If it were the reverse, and UTF-8 were backwards compatible with EUC-JP/CJK, but not ASCII, I doubt you'd be pushing fo…

There is no character in EUC-JP or Shift-JIS that is not in Unicode--the explicit goal of Unicode in its original formulation was to be able to losslessly round-trip any other charset through Unicode, and the initial version of Unicode incorporated the source kanji lists for the EUC-JP/Shift-JIS charsets in their entirety.

The obvious problem for Shift-JIS roundtripping is 0x5C (is it ¥ or \), not the kanji.

Re: Unicode Is Awesome

#122

Post by [deleted] »

RcouF1uZ4gsC wrote: post #15

> Unicode is simply a 16-bit code - Some people are under the misconception that Unicode is simply a 16-bit code where each character takes 16 bits and therefore there are 65,536 possible characters. This is not, actually, correct. It is the single most common myth about Unicode, so if you thought that, don't feel bad. Verity Stob has a great column https://www.theregister.co.uk/2013/10/04/verity_stob_unicode... wher…

[deleted]

Re: Unicode Is Awesome

#123

Post by tomp »

jrochkind1 wrote: post #54
ncmncm wrote: post #48

Earlier quoted context omitted.

There is way more than enough wacky stuff introduced by Unicode. Having dozens of letters A, for example. And giving a Japanese Kanji character the same code as a Chinese one that usually looks similar.

Unicode did not introduce having dozens of letters A, they existed without unicode. Unicode just gives you a way to represent them -- and bonus, often to normalize them all to a normal letter A too. It is a mistake to think that Unicode has the ability to people's text behavior by not supporting things. I mean, maybe it does now that it's so popular, but in order to get adoption it had to support what people were act…

There's no way to distinguish "A" "uppercase a" and "Α" "uppercase α" in written text, but they're different Unicode letters (and might be rendered differently depending on font).

Re: Unicode Is Awesome

#124

Post by tomp »

Shorel wrote: post #69
sansnomme wrote: post #63

Earlier quoted context omitted.

Yeah, that is quite inconsistent. Kanji literally means "Chinese Character" so it should be the same for the letter A. Unless a French A isn't equivalent to an English A.

Cyrillic А is not the same as English A. For example, some fonts render A in a way that looks like Cyrilic Л. (Like The Mandalorian title screen.) This would be incorrect if using the same A for both: https://i.ytimg.com/vi/V8fC7bdV-mI/maxresdefault.jpg

If the readers were to confuse A and Д, that's a problem with the font, not the letters. Cyrillic, Greek and Latin A are all one letter (in uppercase).

Re: Unicode Is Awesome

#125

Post by tomp »

lifthrasiir wrote: post #96
happytoexplain wrote: post #46

Earlier quoted context omitted.

I'm in love with Swift's approach, where the default representation is a well defined thing that both users and developers think of as "characters", but all the other representations are trivially accessible.

I disagree. Grapheme clusters are locale-dependent, much like string collation is locale-dependent. What Unicode gives you by default, the (extended) grapheme cluster, is as useful as the DUCET (Default Unicode Collation Element Table); while you can live with them, you would be unsatisfied. In fact there are tons of Unicode bugs that can't be corrected due to the compatibility reason, and can only be fixed via tailo…

Can you give examples of locale-dependent things, or issues with extended grapheme clusters?

Re: Unicode Is Awesome

#126

Post by msla »

msla wrote: post #91
zzo38computer wrote: post #31

Earlier quoted context omitted.

UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text). But I was not talking about the encoding; I was talking about the Unicode character set.

> UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text). This is utterly incoherent.

Can anyone explain how the statement I responded to makes sense?

I must be wrong, getting so many disagreements.

Re: Unicode Is Awesome

#127

Post by throw0101a »

pdonis wrote: post #70
userbinator wrote: post #40

Earlier quoted context omitted.

If you extend ASCII to CP1252, which is the most common encoding besides/before UTF-8 became common, then you do get those accented characters (and that's likely responsible for the popularity of '1252.) In fact, the first 256 characters of Unicode are almost identical to CP1252. I'm pretty sure that's not a coincidence.

> the first 256 characters of Unicode are almost identical to CP1252. I'm pretty sure that's not a coincidence. That depends on whether you consider the fact that Windows CP 1252 is almost identical to Latin-1 (ISO-8859-1), which is exactly the first 256 characters of Unicode, to be a coincidence.

> This character encoding is a superset of ISO 8859-1 in terms of printable characters, but differs from the IANA's ISO-8859-1 by using displayable characters rather than control characters in the 80 to 9F (hex) range.

* https://en.wikipedia.org/wiki/Windows-1252

Re: Unicode Is Awesome

#128

Post by tialaramex »

jakeogh wrote: post #120
tialaramex wrote: post #111

Earlier quoted context omitted.

More particularly: Presentation variant is not a justification for inclusion in Unicode BUT prior encoding in another character set is. Unicode sets a high priority on roundtripping. The idea is that if you take some data in any one character set X and convert it to Unicode, you should preserve all the meaning by doing this, such that you could losslessly convert it back to encoding X. It's like the wordprocessor pro…

Is there a realitvely easy way to find the character set that was included for uppercase superscript W? (ᵂ)

In the case of U+1D42 Modifier Letter Capital W I was wrong about the cause, it was in fact specifically added on the rationale that for this purpose (phonetics) the presentation was semantic in nature, and so the plain text (thus Unicode) needed to preserve these symbols which could otherwise be handled by a presentation layer.

U+1D42 Modifier Letter Capital W was added in Unicode 4.0 as part of the Phonetic Extensions and Wikipedia provides a long list of Unicode committee paperwork regarding this: https://en.wikipedia.org/wiki/Phonetic_Extensions

You can see that initially it would have been numbered differently and then over the course of several drafts the proposal evolved until it was assigned U+1D42

Re: Unicode Is Awesome

#129

Post by lifthrasiir »

tomp wrote: post #125
lifthrasiir wrote: post #96

Earlier quoted context omitted.

I disagree. Grapheme clusters are locale-dependent, much like string collation is locale-dependent. What Unicode gives you by default, the (extended) grapheme cluster, is as useful as the DUCET (Default Unicode Collation Element Table); while you can live with them, you would be unsatisfied. In fact there are tons of Unicode bugs that can't be corrected due to the compatibility reason, and can only be fixed via tailo…

Can you give examples of locale-dependent things, or issues with extended grapheme clusters?

Hangul normalization and collation is broken in Unicode, albeit for slightly different reasons. The Unicode Collation Algorithm explictly devotes two sections related to Hangul; the first section, for "trailing weights" [1], is recommended for the detailed explanation.

The Unicode Text Segmentation standard [2] explicitly mentions that Indic aksaras [3] require the tailoring to grapheme clusters. Depending on the view, you can also consider orthographic digraphs as examples (Dutch "ij" is sometimes considered a single character for example).

[1] https://www.unicode.org/reports/tr10/#Trailing_Weights

[2] https://unicode.org/reports/tr29/

[3] https://en.wikipedia.org/wiki/Aksara#Grammatical_tradition

Re: Unicode Is Awesome

#130

Post by paulddraper »

tomp wrote: post #125
lifthrasiir wrote: post #96

Earlier quoted context omitted.

I disagree. Grapheme clusters are locale-dependent, much like string collation is locale-dependent. What Unicode gives you by default, the (extended) grapheme cluster, is as useful as the DUCET (Default Unicode Collation Element Table); while you can live with them, you would be unsatisfied. In fact there are tons of Unicode bugs that can't be corrected due to the compatibility reason, and can only be fixed via tailo…

Can you give examples of locale-dependent things, or issues with extended grapheme clusters?

For example, the text "ch" (U+0063 U+0068) is two grapheme clusters in English contexts, but one grapheme cluster in Czech contexts, collated between "h" and "i". [1]

According to Unicode, the text "Chemie" is written exactly the same whether it's the German or the Czech word. However, a German will say it has six letters and a Czech will say it has five.

Unicode provided a unified way to express international characters within the same text, but the context (i.e. locale) external to the text is still required to sensibly collate and manipulate it according to human sensibilities.

The default definition of grapheme clusters is simply a compromise for a global, locale-less understanding of collation/manipulation of Unicode characters.

> The Unicode definitions of grapheme clusters are defaults: not meant to exclude the use of more sophisticated definitions of tailored grapheme clusters where appropriate. Such definitions may more precisely match the user expectations within individual languages for given processes. For example, “ch” may be considered a grapheme cluster in Slovak, for processes such as collation. The default definitions are, however, designed to provide a much more accurate match to overall user expectations for what the user perceives of as characters than is provided by individual Unicode code points.

> Note: The default Unicode grapheme clusters were previously referred to as "locale-independent graphemes." The term cluster is used to emphasize that the term grapheme is used differently in linguistics. For simplicity and to align terminology with Unicode Technical Standard #10, “Unicode Collation Algorithm” [UTS10], the terms default and tailored are preferred over locale-independent and locale-dependent, respectively.

[1] https://en.wikipedia.org/wiki/Ch_(digraph)

[2] http://www.unicode.org/reports/tr29/

Post reply on HN