What every software developer must know about Unicode in 2023

Post by sdrothrock » Tue, Oct 03, 2023, 11:43 AM UTC

ufo wrote: post #469

sdrothrock wrote: post #147

Earlier quoted context omitted.

> they were in some old Japanese character set This implies that they're obsolete, but they're not -- they're still in very common use today. You can type them in Japanese by typing まる (maru, circle) and the number, then pick it out of the IME menu. Some IMEs will bring them up if you just type the number and go to the menu, too. :)

What do the Japanese use the circled numbers for?

Post by layer8 » Tue, Oct 03, 2023, 12:16 PM UTC

jameshart wrote: post #403

layer8 wrote: post #290

Earlier quoted context omitted.

In terminals there is a distinction between single-width and double-width characters (east-asian characters, in particular). E.g. the three characters A美C would take up the width of four ASCII monospace characters, the “美” being double-width. Similarly, for composed characters like say the ligature “ff”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).

Unicode even has distinct full width and half width variants of Japanese katakana - where ‘full width’ is (in theory) as wide as two Latin characters. Latin: Katakana Full width: カタカナ Half width: ｶﾀｶﾅ (How that fixed width text looks in a web browser is anyone’s guess though. On iOS none of the Japanese kana stay on the fixed grid.)

Post by archgoon » Tue, Oct 03, 2023, 12:25 PM UTC

grumpyprole wrote: post #457

jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

> And for this reason, String iteration should be based on codepoints Why not offer both and be clear about it? Rather than just "length", why not call them code points? The Python docs for "len" which can be called on a unicode string say "Return the length (the number of items) of an object.". It doesn't look like a clear and easy to use API to me.

  for c in "Hello":
    pass

  if word[0] == 'H':
     pass

Post by nabla9 » Tue, Oct 03, 2023, 12:27 PM UTC

nottorp wrote: post #275

nabla9 wrote: post #141

>3 Grapheme Cluster Boundaries >It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a use…

> These user-perceived characters are approximated by what is called a grapheme cluster, which can be determined programmatically. From everything i've read or heard about unicode, "determined programmatically" is false?

Post by jandrese » Tue, Oct 03, 2023, 12:41 PM UTC

PeterisP wrote: post #395

JonChesterfield wrote: post #91

Prior to this article, I knew graphemes were a thing and that proper unicode software is supposed to count those instead of bytes or code points. I didn't know that unicode changes the definition of grapheme in backwards incompatible fashion annually, so software which works by grapheme count is probably inconsistent with other software using a different version of the standard anyway. I'm therefore going to continue…

Comparing by memcmp will result in false negatives unless you can ensure that all incoming text gets normalized to a particular canonical form.