What every software developer must know about Unicode in 2023 • Hacker News Board

Re: What every software developer must know about Unicode in 2023

#291

Post by mostlylurks » Mon, Oct 02, 2023, 7:36 PM UTC

School-Cotton wrote: post #231

Findecanor wrote: post #217

Earlier quoted context omitted.

Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

The nordic layout(s) offer such a mechanism to allow people to type in letters that you'll find in various other European languages, even though the extra letters used in the languages themselves (ÅÄÖÆØ) are present as their own keys. Interestingly, the Swedish layout has no dedicated é key, although é occurs in some Swedish words.

Re: What every software developer must know about Unicode in 2023

#292

Post by TacticalCoder » Mon, Oct 02, 2023, 7:37 PM UTC

> For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points!

It's a poor and misleading example for it is definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there (french is the language where 'é' is the most common).

'é' is U+00F9, one codepoint, definitely not two.

Now you could say: but it is also the two codepoints one. But that's precisely what makes Unicode the complete, total and utter clusterfuck that it is.

And hence even an article explaining what every programmer should know about Unicode cannot even get the most basic example right. Which is honestly quite ironic.

Re: What every software developer must know about Unicode in 2023

#293

Post by lucideer » Mon, Oct 02, 2023, 7:37 PM UTC

TRiG_Ireland wrote: post #262

lucideer wrote: post #129

Earlier quoted context omitted.

As an Irish person, while we have en_IE which is great (and solves most of the problems you list re: Euro-centric defaults + English), I'd still quite like to have an even more broad / trans-language / "cosmopolitan" locale to use. I mainly type in English but occasionally other languages - I use a combination of Mac & Linux - macOS has an (off-by-default but enable-able) lang-changer icon in the tray that is handy e…

> US English spellings & conventions are quite a departure from other dialects. As far as the written, formal language is concerned, English really has only three dialects: US American, Canadian, and everywhere else. There are some other subtle differences (such as "robots" for traffic lights in South Africa, or "minerals" for fizzy drinks in Ireland¹), but that's pretty much it. ¹ Yes, this isn't just slang in Irela…

> As far as the written, formal language is concerned, English really has only three dialects

This is true, but I don't see why the "formal" qualifier is needed here :) There are much more than 3 dialects of English, both written & spoken.

Especially there's a fair few extremely common notable differences in (casual, written) Irish English: the word "amn't" (among other less common contractions), the alternative present tense of the verb "to be" (i.e. "do be"), various regional plurals of "you", and - perhaps the most common - prepositional pronouns, etc. etc.

Re: What every software developer must know about Unicode in 2023

#294

Post by mostlylurks » Mon, Oct 02, 2023, 7:39 PM UTC

riggsdk wrote: post #271

School-Cotton wrote: post #231

Earlier quoted context omitted.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Danish keyboards also require you to press '¨' first and then 'o' to produce 'ö'.

Do the danes not have the mechanism that is found on Finnish keyboard layouts, where pressing AltGr+Ö yields Ø and AltGr+Ä yields Æ, except in reverse?

Re: What every software developer must know about Unicode in 2023

#295

Post by favorited » Mon, Oct 02, 2023, 7:40 PM UTC

thyselius wrote: post #202

Wonderful to learn more about Unicode. Does anyone know how to write a function (preferably in swift) to remove emoji? This is surprisingly hard (if the string can be any language, like English or Chinese). There’s been multiple attempts on Stackoverflow but they’re all missing some of them, as Unicode is so complex.

Here's a 1-liner, producing the string "text 0123 漢字":

`String("text EMOJI 0123 漢字".unicodeScalars.filter({ !$0.properties.isEmojiPresentation }))`

(I've had to substitute EMOJI for a smiley face, because HN is bad at text encoding.)

Re: What every software developer must know about Unicode in 2023

#296

Post by Sardtok » Mon, Oct 02, 2023, 7:40 PM UTC

riggsdk wrote: post #271

School-Cotton wrote: post #231

Earlier quoted context omitted.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Danish keyboards also require you to press '¨' first and then 'o' to produce 'ö'.

But do you really use ö much over ø?

Re: What every software developer must know about Unicode in 2023

#297

Post by WorldMaker » Mon, Oct 02, 2023, 7:44 PM UTC

arp242 wrote: post #278

WorldMaker wrote: post #267

Earlier quoted context omitted.

> The combinations aren't infinite here. They certainly are. Languages are a creative space driven by the human imagination. Give people enough time and they'll build new combinations for fun or for profit or for research or for trying to capture a spoken word/tone poem in just the right sort of exciting way. You may frown on "Zalgo text" [1] (and it is terrible for accessibility), but it speaks to a creative mood or…

Well, in theory it's infinite, but in reality it's not of course. We've got 150K assigned codepoints assigned, leaving us with 950K unassigned codepoints. There's truly massive amounts of headroom. To be honest I think this argument is rather too abstract to be of any real use: if it's a theoretical problem that will never occur in reality then all I can say is: . But like I said: I'm not "against" combining marks, p…

The Unicode committees have addressed this for languages such as Latin, Cyrillic, and others and stated outright that decomposed forms should be preferred and decomposition canonical forms are generally the safest for interoperability and operations such as collation (sorting) and case folding (lowercase to uppercase transformations).

Unicode can't get rid of the many precombined characters for a huge number of backward compatibility reasons (including compatibility with ancient Mainframe encodings such as EBCDIC which existed before computer fonts had ligature support), but they've certainly done what they can to suggest the "normal" forms in this decade should "prefer" the decomposed combinations.

> If I want to propose something like a "%" mark on top of the "e" to indicate, I don't know, something, then I can't do that regardless of whether combining characters are used

This is where emoji as a living language actually shines a living example: It's certainly possible to encode your mark today as a ZWJ sequence, say «e ZWJ %», though you might want to consider for further disambiguation/intent-marking adding a non-emoji variation selector such as Variation Selector 1 (U+FE00) to mark it as "Basic Latin"-like or "Mathematical Symbol"-like. You can probably get away with prototyping that in a font stack of your choosing using simple ligature tools (no need for private-use encodings). A ZWJ sequence like that in theory doesn't even "need" to ever be standardized in Unicode if you are okay with the visual fallback to something like "e%" in fonts following Unicode standard fallback (and maybe a lot of applications confused by the non-recommended grapheme cluster). That said, because of emoji the process for filing new proposals for "Recommended ZWJ Sequences" is among the simplest Unicode proposals you can make. It's not entirely as Catch-22 on "needs to have seen enough usage in written documents" as some of the other encoding proposals.

Of course, all of that is theory and practice is always weirder and harder than theory. Unicode encoding truly living languages like emoji is a blessing and it does enable language "creativity" that was missing for a couple of decades in Unicode processes and thinking.

Re: What every software developer must know about Unicode in 2023

#298

Post by danbruc » Mon, Oct 02, 2023, 7:46 PM UTC

masklinn wrote: post #131

danbruc wrote: post #114

Earlier quoted context omitted.

Four. Obviously. The more interesting question is whether the Unicode rules actually give that answer. EDIT: Just checked it using the first online tool [1] that came up and it indeed says four. So all is good. [1] https://onlinetools.com/unicode/extract-unicode-graphemes

It should be 4 as long as you count the grapheme clusters which is what e.g. Swift does (hence String#count being O(n)). In Javascript, you can get the same information through Intl.Segmenter, segments by grapheme cluster by default.

You could also have it in O(1), just store and maintain it as you usually store the length in bytes or code units. If you had all your string operations like substring work with grapheme clusters by default, which might arguably make sense quite often, then that could actually be a good decision. It might even make sense to maintain a list with pointers to each grapheme cluster or of all the grapheme cluster lengths together with the actual string data. Or maybe not, would probably depend heavily on the workload.

Re: What every software developer must know about Unicode in 2023

#299

Post by mplewis » Mon, Oct 02, 2023, 7:46 PM UTC

spacechild1 wrote: post #268

toastercat wrote: post #119

Earlier quoted context omitted.

Anytime tonsky's site gets posted here, I'm reminded by how awful it is, which is ironic given his UI/UX background. The site's lightmode is a blinding saturated yellow, and if you switch into darkmode, it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.

It is obviously a joke (and a good one, I dare say). The fact that people seem to take it seriously says something about the contemporary state of webdesign :)

It would be a better joke if there were an option to turn the joke off. As it is, dark mode doesn't exist and the pointers occlude text.

Re: What every software developer must know about Unicode in 2023

#300

Post by cyxxon » Mon, Oct 02, 2023, 7:52 PM UTC

gumby wrote: post #263

WorldMaker wrote: post #255

Earlier quoted context omitted.

> So the ﬁ ligature was in a legacy encoding system and thus must be in Unicode. Most of the pre-composed latin ligatures are generally from EBCDIC codepages. People in the ancient Mainframe era wanted nice typesetting too, but computer fonts with ligature support were a much later invention. You can see ﬁ and several others directly in EBCDIC code page 361: https://en.wikibooks.org/wiki/Character_Encodings/Code_Tabl…

Thanks. Some alphabets have precomposed ligatures that aren't really letters, like old German alphabets with tz, ch, ss (I only know how to type the last one, ß, because the others have died out over the last hundred years). Actually in German (at least) ä, ö and ü really are actually ligatures for ae, oe, and ue -- the scribes started to write the E's on their sides above the base letters, and over time the superscr…

That's sounds a bit false to me. The Umlaute (ä,ö, ü) and the "eszett" ß are actually part of the German alphabet[1]. Also it is kinda weird to describe them as ligatures of the original letters and the diaeresis, because while this is what they started out as a long time ago, they are just their own letters now (as opposed to "real" stylistic ligatures like combining fi into one glyph). The advice your kid was told that they can be replaced with ae, oe and ue is correct - it is a replacement nowadays.

[1] https://de.wikipedia.org/wiki/Deutsches_Alphabet