What every software developer must know about Unicode in 2023 • Hacker News Board

Re: What every software developer must know about Unicode in 2023

#271

Post by riggsdk » Mon, Oct 02, 2023, 7:03 PM UTC

School-Cotton wrote: post #231

Findecanor wrote: post #217

Earlier quoted context omitted.

Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Danish keyboards also require you to press '¨' first and then 'o' to produce 'ö'.

Re: What every software developer must know about Unicode in 2023

#272

Post by rdtsc » Mon, Oct 02, 2023, 7:06 PM UTC

> The only modern language that gets it right is Swift:

    print("...".count)
    // => 1

And Erlang/Elixir! I guess they are not "cool" enough. But they correctly interpret that as one grapheme cluster.

    % erl +pc unicode
    > string:length("...").
    1

(... here is the U+1F926 U+1F3FB U+200D U+2642 U+FE0F emoji)

Re: What every software developer must know about Unicode in 2023

#273

Post by zackmorris » Mon, Oct 02, 2023, 7:08 PM UTC

The only modern language that gets it right is Swift:

Apple did a fairly good job with unicode string handling starting in Cocoa and Objective-C, by providing methods to get the number of code points and/or bytes:

https://stackoverflow.com/questions/15582267/cfstring-count-...

I feel that this support of both character count and buffer size in bytes is probably the way to go. But Python 3 went wrong by trying to abstract it away with encodings that have unintuitive pitfalls that broke compatibility with Python 2:

https://blog.feabhas.com/2019/02/python-3-unicode-and-byte-s...

There's also the normalization issue. Apple goofed (IMHO) when they used NFD in HFS+ filenames while everyone else went with NFC, but fixed that in APFS:

https://unicode.org/faq/normalization.html

https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...

Re: What every software developer must know about Unicode in 2023

#274

Post by WorldMaker » Mon, Oct 02, 2023, 7:14 PM UTC

sebstefan wrote: post #150

nabla9 wrote: post #141

>3 Grapheme Cluster Boundaries >It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a use…

Oh my god, is there ever anything simple about unicode

Compared to the ancient world of EBCDIC versus ASCII versus various ISO standards versus country-defined encodings versus Extended EBCDIC code pages versus Extended ASCII code pages which varied depending on operating system, nearest flag pole, network adapter, time of day, etc…: Unicode will forever be a simpler walk in the park.

It's complexity is a relief compared to where we've been. It's definitely not simple, but it will forever be far simpler than what our grandmothers had to work with if they were writing international software.

Re: What every software developer must know about Unicode in 2023

#275

Post by nottorp » Mon, Oct 02, 2023, 7:14 PM UTC

nabla9 wrote: post #141

>3 Grapheme Cluster Boundaries >It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a use…

> These user-perceived characters are approximated by what is called a grapheme cluster, which can be determined programmatically.

From everything i've read or heard about unicode, "determined programmatically" is false?

Re: What every software developer must know about Unicode in 2023

#276

Post by hot_gril » Mon, Oct 02, 2023, 7:15 PM UTC

hot_gril wrote: post #143

dathinab wrote: post #46

> The only modern language that gets it right is Swift: I disagree. What is the "right" things is use-case dependent. For UI it's glyph bases, kinda, more precise some good enough abstraction over render width. For which glyphs are not always good enough but also the best you can get without adding a ton of complexity. But for pretty much every other use-case you want storage byte size. I mean in the UI you care abou…

Swift made an effort to handle grapheme clusters but severely over-complicated strings by exposing performance details to users. Look at the complex SO answers to what should be simple questions, like finding a substring: https://news.ycombinator.com/item?id=32325511 , many of which changed several times between Swift versions I was working on an app in Swift that needed full emoji support once. Team ended up writing…

Also, realized "needed full emoji support" sounds silly. It needed to do a lot of string manipulation, with extended grapheme clusters in mind, mainly for the purpose of emojis.

Re: What every software developer must know about Unicode in 2023

#277

Post by layer8 » Mon, Oct 02, 2023, 7:16 PM UTC

riggsdk wrote: post #270

jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

There are libraries that help with iterating both code-points and grapheme clusters... - but are there any of them that can help decide what to do for example when pressing backspace given an input string and a cursor position? Or any other text editing behavior. This use-case-dependent behavior must have some "correct" behavior that is standardized somewhere? Like a way to query what should be treated like a single…

Behavior that depends on whether you edited something else in between, or that depends on timing, is just bad. Either always backspace grapheme clusters, or else backspace characters, possibly NFC-normalized. I could also imagine having something like Shift+Backspace to backspace NFKD-normalized characters when normal Backspace deletes grapheme clusters.

As for selection and cursor movement, grapheme clusters would seem to be the correct choice. Same for Delete. An editor may also support an “exploded” view of separate characters (like WordPerfect Reveal Codes) where you manipulate individual characters.

Re: What every software developer must know about Unicode in 2023

#278

Post by arp242 » Mon, Oct 02, 2023, 7:16 PM UTC

WorldMaker wrote: post #267

arp242 wrote: post #90

Earlier quoted context omitted.

> so the idea of a world with only precomposed letter forms is more of a exponential explosion in the character set "Exponential explosion" is really putting it too strong; it's perfectly possible to just add ǿ and á̤ and a bunch of other things. The combinations aren't infinite here. The problem with e.g. Latin script isn't necessarily that combining characters exist, but that there's two ways to represent many th…

> The combinations aren't infinite here. They certainly are. Languages are a creative space driven by the human imagination. Give people enough time and they'll build new combinations for fun or for profit or for research or for trying to capture a spoken word/tone poem in just the right sort of exciting way. You may frown on "Zalgo text" [1] (and it is terrible for accessibility), but it speaks to a creative mood or…

Well, in theory it's infinite, but in reality it's not of course.

We've got 150K assigned codepoints assigned, leaving us with 950K unassigned codepoints. There's truly massive amounts of headroom.

To be honest I think this argument is rather too abstract to be of any real use: if it's a theoretical problem that will never occur in reality then all I can say is: .

But like I said: I'm not "against" combining marks, purely in principle it's probably better, I'm mostly against two systems co-existing. In reality it's too late to change the world to decomposed (for Latin, Cyrillic, some others) because most text already is pre-composed, so we should go full-in on pre-composed for those. With our 950k unassigned codepoints we've got space for literally thousands of years to come.

Also this is a problem that's inherent in computers: on paper you can write anything, but computers necessarily restrict that creativity. If I want to propose something like a "%" mark on top of the "e" to indicate, I don't know, something, then I can't do that regardless of whether combining characters are used, never mind entirely new characters or marks. Unicode won't add it until it sees usage, so this gives us a bit of a catch-22 with the only option being mucking about with special fonts that use private-use (hoping it won't conflict with something else).

Re: What every software developer must know about Unicode in 2023

#279

Post by pests » Mon, Oct 02, 2023, 7:19 PM UTC

arp242 wrote: post #245

d3w4s9 wrote: post #214

Earlier quoted context omitted.

The first time I heard that Unicode would support emoji, I knew it would be a recipe for disaster. And I definitely was not disappointed.

I mean, I don't dislike the concept personally. I actually really hate how HN strips them. But the technical implementation? Yeah, that could have gone a lot better IMHO. One must also wonder if some things really had to be added in the first place, e.g. for people kissing it's: (person|man|woman)(skin-tone)? ZWJ ZWJ ZWJ (person|man|woman)(skin-tone)? This is NOT a complaint about that they added diversity as such, i…

> there's 98 combinations to choose from here.

Less than that since a default skin color can be set in most apps. I'm sure setting a gender will come soon so the entire first part of that emoji can be auto-guessed. Then its just showing the other options in the UI. Really all of this is UI design as even with the 98 combinations you can still display it as 4/5 options you drill down.

> who finds it convenient to pick all the correct genders and skin tones from the UI for both people?

I just checked and searching "kissing" in my iOS emoji keyboard inside Messenger showed just 4 of the emoji's your describing - defaulting both skin tones to my settings and then the four M/F pair ups. Plus some non-related kissing emojis like the cat kissing.

Re: What every software developer must know about Unicode in 2023

#280

Post by WalterBright » Mon, Oct 02, 2023, 7:20 PM UTC

Quotes from the article illustrating what a train wreck Unicode has become:

"The problem is, in Unicode, some graphemes are encoded with multiple code points!"

"An Extended Grapheme Cluster is a sequence of one or more Unicode code points that must be treatead as a single, unbreakable character."

"Starting roughly in 2014, Unicode has been releasing a major revision of their standard every year."

"Å" === "Å" "Å" === "Å" "Å" === "Å" What do you get? False? You should get false, and it’s not a mistake.

"That’s why we need normalization."

"Unicode is locale-dependent"

The article forgot one: characters that switch presentation to right-to-left.