Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a comment on Hacker News led to 4½ new Unicode characters

unicodepowersymbol.com

Post reply on HN
Print view View original

311–320 of 429 posts

Re: How a comment on Hacker News led to 4½ new Unicode characters

#311

Post by msbarnett »

dalke wrote: post #231
xyproto wrote: post #78

Earlier quoted context omitted.

Being able to easily check if a letter is between 'a' and 'z' in code is an advantage. This is only possible if the codepoints are sequential.

It's not a big advantage. EBCDIC, for example, didn't do that, and programmers managed just fine without it. Also, why are you doing that check? Is it to see if something is lowercase? If so, your check will get the wrong answer for lowercase letters like å. Unicode does have a way to check if something is uppercase/lowercase, when that distinction exists. This is in UnicodeData.txt.

> It's not a big advantage. EBCDIC, for example, didn't do that, and programmers managed just fine without it.

You might try asking an old IBM programmer just how "fine" they felt dealing with EBCDIC...

Re: How a comment on Hacker News led to 4½ new Unicode characters

#312

Post by igravious »

coldtea wrote: post #189
igravious wrote: post #129

Earlier quoted context omitted.

How are traffic signs not in "running text" in books about the rules of the road and such like?

Running text means INSIDE text (as in: "running along" with the other characters), not "used in a book as illustration".

Yes, of course. What was I thinking?

Re: How a comment on Hacker News led to 4½ new Unicode characters

#313

Post by msbarnett »

thaumasiotes wrote: post #294
footpath wrote: post #276

Earlier quoted context omitted.

It is true to some extent. While 青 and 靑 have different codepoints, there are plenty of characters of the same codepoint that are rendered differently depends on the language specificed: https://en.wikipedia.org/wiki/Han_unification#Examples_of_la... Han characters that are traditionally viewed as variants of one another, or that are simplified from more complex logograms (such as 龜, which was simplified into 亀 in Ja…

I do know about the issue; it causes problems for me. But I couldn't let the claim that all semantically equivalent characters were unified pass. > the stylistically different ones usually belong to the same codepoint Fair enough. Do you happen to know why 青 and 靑 weren't unified?

Han Unification "rules" were an inconsistent mess, but I do know that in Japanese 靑 was at one time a printer's simplification of 青, so you could find either in texts, and the Consortium tended to encode a character separately if you could find printed examples of both in the same language.

Re: How a comment on Hacker News led to 4½ new Unicode characters

#314

Post by kens »

copperx wrote: post #249
kens wrote: post #101

Earlier quoted context omitted.

Unicode is supposed to include symbols that appear in "running text", not standalone icons. So no on traffic signs for instance. (There are exceptions for historical reasons. And emoji are a totally separate story.)

I have never read a book that had a snowman in the running text, so what's the story for emoji?

Emoji were added to Unicode for compatibility with various mobile phones, so they would have a standard encoding. That's how Unicode ended up with the poop emoji for example - they didn't sit around thinking "what we really need is...". Since people really, really want more emoji, Unicode is sort of stuck constantly adding more. If you want to propose new emoji, the rules are at http://www.unicode.org/emoji/selection.html

Text symbols (as opposed to emoji) have different rules. Basically, the symbol needs to be used in "running text" (i.e. normal text), like "containers with [recycling symbol] can be recycled" or "he bid 2[club]". Traffic signs for example are not normally used in the middle of text, so they aren't encoded in Unicode. To get the Bitcoin symbol encoded, I needed to show that it was used in text, not just as a standalone icon. The full rules for symbols in Unicode are at http://www.unicode.org/pending/symbol-guidelines.html

For the snowman in particular, it was added to Unicode because it was a symbol used in the character set for Japanese TV broadcasts, see http://www.unicode.org/L2/L2007/07391-n3341.pdf

TL;DR: Don't argue "Why does Unicode have a poop emoji but no symbol for X?" - the rules are totally different for emoji and symbols.

Edit: does HN strip out arbitrary Unicode characters now? I originally had Unicode characters in place of [recycling symbol] and [club], but they disappeared when I submitted.

Re: How a comment on Hacker News led to 4½ new Unicode characters

#315

Post by WalterBright »

merijnv wrote: post #167
WalterBright wrote: post #158

Earlier quoted context omitted.

There's also precedent. Continental Europe standardized on "STOP" on stop signs back in the 70's, even though no continental language has "stop" in it.

Excuse me? No continental European language has the word stop in it? You should probably learn more languages before making claims like that. Stop is a word in Dutch (the first recognisable use of the word I could find dates back to 1287). And German has stopf. I couldn't find a date for that, because my German isn't good enough to read their etymology dictionary, but it's source is Old High Germanic, so it's safe to…

German stop signs used the word "HALT" before. My German dictionary defines "stopf" as darning yarn, and "stoppen" as stop. Not quite the same spelling. Typing "stop" into French Google translate autocompletes to "stoppé". I wouldn't be in the least surprised if the American spelling crept into many European languages as a result of the sign being ubiquitous for 41 years - it would be surprising if it didn't.

I wouldn't underestimate the influence on the language of the American occupation after the war, either, nor the global influence of American business since the war. English words have crept in everywhere.

Re: How a comment on Hacker News led to 4½ new Unicode characters

#316

Post by msbarnett »

xg15 wrote: post #218
jomamaxx wrote: post #206

Earlier quoted context omitted.

"many visually identical code points" The emoji code points can be represented differently on different systems given their meaning. So it makes sense to have different emojis for different 'meanings'. The 'moon' switch here does no mean 'moon' - it means 'standby' or whatever. It may look noticeably different on different systems. Think from a design perspective: you have 5 emojis to represent 'clouds, sky, earth' e…

But that doesn't explain the inconsistency in the current case. So if a system wants to render "on" differently than "straight vertical line", that's possible. However, if "off" should be rendered differently than "circle", that's not possible. (Or only possible with out-of-band information or modifier characters which would still have to be defined)

Yeah.

It's a mess. If you want to write a document in Japanese that talks about a Chinese character which is written differently than its Japanese version, you can, or can't, achieve this in Unicode, depending on the character, its history, and the mood of the consortium the day it was assigned.

The reality is that Unicode is governed by people, some of those people are grumpy reductionists who push for a minimum of symbols and a maximum of meaning-overloads, and others are more liberal and tend to advocate the opposite, and the result is a compromise, and is in areas very messy.

Re: How a comment on Hacker News led to 4½ new Unicode characters

#317

Post by WalterBright »

function_seven wrote: post #166
WalterBright wrote: post #155

Earlier quoted context omitted.

Why is it easier to infer that 'O' means off than 'OFF' for a person who does not know any English?

Because it's a zero and the | is a one. Arabic numerals are more universal, and the convention of 0 for off and 1 for on was established precisely to avoid picking a language. Then the combined glyph for an on/off button was created, along with the similar broken circle glyph for on/standby. Those have squarish proportions, so the corresponding 0 and 1 glyphs are needed to match those proportions. Hence the four symb…

> the convention of 0 for off and 1 for on was established precisely to avoid picking a language

I figured that politics was the case. What is less supportable is working backwards from that to concoct a rational reason. People who understand digital electronic conventions are highly unlikely to not recognize "ON" and "OFF". Furthermore, anyone who does not know either would find "ON" just as easy to learn as "|".

(That's why I referred to "ON" as a 'glyph'.)

Re: How a comment on Hacker News led to 4½ new Unicode characters

#318

Post by thaumasiotes »

msbarnett wrote: post #307
thaumasiotes wrote: post #300

Earlier quoted context omitted.

Every year you see tons of 福倒 printed too (福, but upside down). That has yet to receive a code point.

You're not wrong, but I don't know if it's a great example. CJK characters are, broadly, an example of the Unicode Consortium trying to be way too reductive about what they'd accept, leading to a lot of bad decisions like Han Unification, which caused a lot of damage and which the Consortium has generally now backed away from and recognized as a bad idea. So, yes, if you look closely at CJK character sets in Unicode,…

Unification is a mistake. But 囍 has nothing to do with unification. Do you think that 福倒 should have a code point? Would it be considered one of the chinese characters (very iffy) or one of the holiday symbols?

The 天书 ( https://en.wikipedia.org/wiki/A_Book_from_the_Sky ), by design, consists solely of chinese characters that don't exist. (Theoretically. A couple of them, by oversight, did exist.) They are still recognizably "chinese characters" by virtue of being composed of the same components. Should they have unicode points?

囍 plainly exists, but has no textual use. Is it more similar to 靑 or to ️U+2764 "heavy black heart"?

Re: How a comment on Hacker News led to 4½ new Unicode characters

#319

Post by msbarnett »

thaumasiotes wrote: post #318
msbarnett wrote: post #307

Earlier quoted context omitted.

You're not wrong, but I don't know if it's a great example. CJK characters are, broadly, an example of the Unicode Consortium trying to be way too reductive about what they'd accept, leading to a lot of bad decisions like Han Unification, which caused a lot of damage and which the Consortium has generally now backed away from and recognized as a bad idea. So, yes, if you look closely at CJK character sets in Unicode,…

Unification is a mistake. But 囍 has nothing to do with unification. Do you think that 福倒 should have a code point? Would it be considered one of the chinese characters (very iffy) or one of the holiday symbols? The 天书 ( https://en.wikipedia.org/wiki/A_Book_from_the_Sky ), by design, consists solely of chinese characters that don't exist. (Theoretically. A couple of them, by oversight, did exist.) They are still recog…

> Unification is a mistake. But 囍 has nothing to do with unification.

I'm not saying it does, I'm saying Unification illustrates the fact that the Consortium's decision-making with respect to CJK has changed over time, has frequently been illogical, and shouldn't be pointed at as an example of anything good or sane or worthy of precedent.

The fact that 囍 has a code point but 福倒 doesn't have a codepoint is another example of the Consortium being unnecessarily reductive and intransigent about CJK.

> Do you think that 福倒 should have a code point?

Yes. If we want to be able to talk about it in text (like now), I want to be able to encode it in a standardized way.

> Should they have unicode points?

I'd lean towards no, as they're one-offs, not something broader that people want to discuss and use in text. But I'd be ok with adding them, too. We're not running out of space. There's no value in making CJK so much harder to interop with than everything else, in general.

Re: How a comment on Hacker News led to 4½ new Unicode characters

#320

Post by dalke »

msbarnett wrote: post #311
dalke wrote: post #231

Earlier quoted context omitted.

It's not a big advantage. EBCDIC, for example, didn't do that, and programmers managed just fine without it. Also, why are you doing that check? Is it to see if something is lowercase? If so, your check will get the wrong answer for lowercase letters like å. Unicode does have a way to check if something is uppercase/lowercase, when that distinction exists. This is in UnicodeData.txt.

> It's not a big advantage. EBCDIC, for example, didn't do that, and programmers managed just fine without it. You might try asking an old IBM programmer just how "fine" they felt dealing with EBCDIC...

EBCDIC had many problems, agreed. One is the non-contiguous range. Another is the many variants (https://en.wikipedia.org/wiki/EBCDIC_code_pages lists 11, x2 for the Euro Update).

But how serious is the problem? How many times do you need to test if a given character is one of the 26 allowed letters of the English alphabet, and where you implement it by testing it against the range?

Typically you write it as "islower_english(c)" once, and be done with it. Is that really hard?

If you do think that's a serious problem, then what of those programmers who need to test for lowercase letters in "España", "München", "Diyarbakır", and "façade"?

Post reply on HN