Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

381–390 of 572 posts

Re: What every software developer must know about Unicode in 2023

#381

Post by Dylan16807 »

hgs3 wrote: post #376
Dylan16807 wrote: post #348

Earlier quoted context omitted.

> These properties are assigned to individual characters, not groups of characters or grapheme clusters. But you need to deal with the whole cluster. You can't just look at the properties on a single combining character and know what to do with it. If the article's saying to iterate one cluster at a time, then if you're doing properties a direct consequence is that you should be looking at the properties of specific…

The Unicode Standard does not specify how character properties should be extracted from a grapheme cluster. Programming languages that define "character" to mean grapheme cluster (like Swift) need to establish their own ad-hoc rules. As others have pointed out in this thread, the article is full of the authors own personal opinions. The author suggests iterating text as grapheme clusters, but fails to consider that t…

> The Unicode Standard does not specify how character properties should be extracted from a grapheme cluster. Programming languages that define "character" to mean grapheme cluster (like Swift) need to establish their own ad-hoc rules.

Right. Which means not just iterating by code point.

> The author suggests iterating text as grapheme clusters, but fails to consider that this breaks tokenizers, e.g. a tokenizer for a comma-separated list [1] won't see the comma as "just a comma" if the value after it begins with a combining character.

I don't think they're talking about tokenizers. It's a general purpose rule.

Also I would argue that a CSV file with non-attached combining characters doesn't qualify as "text".

Re: What every software developer must know about Unicode in 2023

#382

Post by mananaysiempre »

jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

> thing that gets deleted when you hit backspace

Is there a canonical source for this part, by the way? Xi copied the logic from Android[1] (per the issue you linked downthread), which is reasonable given its heritage but seems suboptimal generally, and I vaguely remember that CLDR had something to say about this too, but I don’t know if there’s any sort of consensus on this problem that’s actually written down anywhere.

[1] https://github.com/xi-editor/xi-editor/pull/837

Re: What every software developer must know about Unicode in 2023

#383

Post by everyone »

I'm always gonna point out these overly broad titles assuming "every software developer" is some kind of internetty web dev type. I'm a game dev, I try and never touch strings at all, they are a nightmare data type. Strings in a game are like graphics or audio assets, your game might read them and show them to player, but they should never come anywhere near your code or even be manipulated by it. I dont need to know any of that stuff about Unicode.

Re: What every software developer must know about Unicode in 2023

#384

Post by mananaysiempre »

PeterisP wrote: post #379
riggsdk wrote: post #270

Earlier quoted context omitted.

There are libraries that help with iterating both code-points and grapheme clusters... - but are there any of them that can help decide what to do for example when pressing backspace given an input string and a cursor position? Or any other text editing behavior. This use-case-dependent behavior must have some "correct" behavior that is standardized somewhere? Like a way to query what should be treated like a single…

I'd argue that you must use grapheme clusters for text editing and cursor position, because here are popular characters (like ö you used as example) which can be either one or two codepoints depending on the normalization choice, but the difference is invisible to the user and should not matter to the user, so any editor should behave exactly the same for ö as U+00F6 (LATIN SMALL LETTER O WITH DIAERESIS) and ö as a…

> I'd argue that you must use grapheme clusters for text editing and cursor position

Korean packs syllables into Han-script-like squares, but they are unmistakably composed of alphabetic letters, and are both typed and erased that way (the latter may depend on system configuration), yet the NFC form has only a single codepoint per syllable (a fortiori a single grapheme cluster). Hebrew vowel markings are (reasonably) considered to be part of the grapheme cluster including their carrier letter but are nevertheless erased and deleted separately. In both of those cases, pressing backspace will erase less than pressing shift-left, backspace; that is, cursor movement and backspace boundaries are different.

There are IIRC also scripts that will have a vowel both pronounced and encoded in the codepoint stream after the syllable-initial consonant but written before it; and ones where some parts of a syllable will enclose it. I don’t even want to think how cursor movement works there.

Overall, your suggestion will work for Latin, Cyrillic, Greek, and maybe other nonfancy scripts like Armenian, Ge’ez, or Georgian, but will absolutely crash and burn when used for others.

Re: What every software developer must know about Unicode in 2023

#385

Post by PeterisP »

hgs3 wrote: post #376
Dylan16807 wrote: post #348

Earlier quoted context omitted.

> These properties are assigned to individual characters, not groups of characters or grapheme clusters. But you need to deal with the whole cluster. You can't just look at the properties on a single combining character and know what to do with it. If the article's saying to iterate one cluster at a time, then if you're doing properties a direct consequence is that you should be looking at the properties of specific…

The Unicode Standard does not specify how character properties should be extracted from a grapheme cluster. Programming languages that define "character" to mean grapheme cluster (like Swift) need to establish their own ad-hoc rules. As others have pointed out in this thread, the article is full of the authors own personal opinions. The author suggests iterating text as grapheme clusters, but fails to consider that t…

If some tokenizer of a comma-separated list treats the comma (I'm assuming any 0x2C byte) as "just a comma" even if the value after it begins with a combining character, that's a broken, buggy tokenizer, and one that can potentially be exploited by providing some specifically crafted unicode data in a single field that then causes the tokenizer to misinterpret field boundaries. If you combine a character with something, that's not the same character anymore - it's not equal to that, it's not that separator anymore, and you can't tell that unless/until you look at the following codepoints. If the combined character isn't valid, then either the message should be discarded as invalid or the character replaced with U+FFFD, the Replacement Character, but it should definitely not be interpreted as "just a comma" simply because one part of some broken character matches the ASCII code for a comma.

If anything, your example is an illustration why it's dangerous to iterate over codepoints and not graphemes. Unless you're explicitly tranforming encodings to/from unicode, anything that processes text (not the encoding, but actual text content, like tokenizers do) - should work with graphemes as the basic atomic indivisible unit.

Re: What every software developer must know about Unicode in 2023

#386

Post by koreth1 »

loeg wrote: post #333

Pretty clearly, "every software developer" doesn't need to understand Unicode with this level of familiarity, much like "every programmer" doesn't need to know the full contents of the 114 page Drepper paper. For example, I work on a GUID-addressed object store. Everything is in term of bytes and 128-bit UUIDs. Unicode is irrelevant to everyone on my team, and most adjacent teams. There is lots of software like this.

Glad I'm not the only one who was irked by this, and I do need to know a lot about Unicode for my job!

I believe there actually are topics that every software developer ought to know something about, but this isn't one of them. My list would be things more like, the difference between a constant-time algorithm and a quadratic-time one.

Re: What every software developer must know about Unicode in 2023

#387

Post by pornel »

phforms wrote: post #285

Regarding UTF-8 encoding: “And a couple of important consequences: - You CAN’T determine the length of the string by counting bytes. - You CAN’T randomly jump into the middle of the string and start reading. - You CAN’T get a substring by cutting at arbitrary byte offsets. You might cut off part of the character.” One of the things I had to get used to when learning the programming language Janet is that strings are…

You CAN'T do any of these things in Unicode in general, in all of its encodings.

There's no random access in Unicode. It's a stateful system that requires linear scan.

Re: What every software developer must know about Unicode in 2023

#388

Post by PeterisP »

nottorp wrote: post #246
nwellnhof wrote: post #47

Earlier quoted context omitted.

Name one writing system where you really need character composition. Even if there is one, these special cases should be handled outside of Unicode.

The problem is not that you need character composition for some writing systems. It's that there are no rules that would help with everything having an unique representation internally. Even "put the code points forming the composed character in descending numerical order" would be better than nothing. If it was there from the start. However, the Unicode commitee is too busy adding new emojis to make their standard s…

There are rules for that, Unicode has standards (not only formal, but easily usable in most software libraries) for canonical forms that will collapse all the variations to a single representation.

But, of course, unicode can't define that the standard will cover only the canonical forms, and couldn't do that since the start, as it needed backwards compatibility with various pre-unicode encodings which had mutually incompatible principles, so it needed support for both composed and decomposed versions of the same characters.

Re: What every software developer must know about Unicode in 2023

#389

Post by arp242 »

WorldMaker wrote: post #346
arp242 wrote: post #324

Earlier quoted context omitted.

> The Unicode committees have addressed this for languages such as Latin, Cyrillic, and others and stated outright that decomposed forms should be preferred Yes, and that only makes things worse since the overwhelming majority of documents (99.something% last time I checked) uses pre-composed. Also AFAIK just about everyone just ignores that recommendation. This is a classic "reality should adjust to the standard" ty…

> Yes, and that only makes things worse since the overwhelming majority of documents (99.something% last time I checked) uses pre-composed. It shouldn't matter what's in the wild in documents. That's why we have normalization algorithms and normalization forms. Unicode was built for the ugly reality of backwards compatibility and that you can't control how people in the past wrote. These precomposed characters largel…

The thing with normalization is that it's not free, and especially for embedded use cases people seem quite opposed to this. IIRC it requires about ~100K of binary size, ~20K of memory, and some non-zero number of CPU cycles. This is negligible for your desktop computer, but for embedded use cases this matters (or so I've been told).

This comes up in specifications that have a broad range of use cases; when I was involved in this my idea was to just spec things so that there's only one allowed form; you'll still need a small-ish table for this, but that's fine. But that's currently hard because for a few newer Latin-adjacent alphabets some letters cannot be represented without a combining character.

So then you have either the "accept that two things which seem visually similar are not identical" (meh) or "exclude embedded use cases" (meh).

I never really found a good way to unify these use cases. I've seen this come up a few times in various contexts over the years.

> Posted to HN several times has been the well documented proposal process from start to finish (it succeeded) of getting common and somewhat less common power symbols encoded in Unicode.

Would this work for an entirely new symbol I invent today? It's not really the Unicode people that are "difficult" here as such, they just ask for demonstrated usage, which is entirely reasonable, and that's hard to get (or: harder than it was before computers) especially for casual usage. I'm sure that if some country adopts/invents a new script today, as seems to be happening in West-Africa at in recent years, the Unicode people are more than amendable to work with that, but "I just like ‽" is a rather different type of thing.

Re: What every software developer must know about Unicode in 2023

#390

Post by hunter2_ »

riggsdk wrote: post #374
posix86 wrote: post #329

Earlier quoted context omitted.

But typing "ö" (e.g. swiss keyboard) and pressing delete & getting an o would be annoying af

I realize that the editor would be the system to keep track of how the character was entered for this to work. If you made the character from a single keypress it would only make sense that backspace also undid the entire character. Only if you created the character from multiple keypresses it would make sense to "undo" only part of it with backspace (at least until you move away from the character).

> make sense to "undo" only part of it with backspace

I'm not sure that ever really makes sense: it would be a misnomer if "backspace" didn't bring you "back" some amount of horizontal "space," I reckon. This logic holds up not only for cases like ö and emoji (where I'd hope the whole grapheme disappears), but also for scenarios like if one types and an ligature appears, where I'd hope only the disappears: that's fine because you are still going back some space.

If the key ever gets repurposed from "backspace" to "undo" then I would agree that it should step back to the previous state with as much granularity as possible.

Post reply on HN