Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

321–330 of 572 posts

Re: What every software developer must know about Unicode in 2023

#321

Post by gumby »

riggsdk wrote: post #270
jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

There are libraries that help with iterating both code-points and grapheme clusters... - but are there any of them that can help decide what to do for example when pressing backspace given an input string and a cursor position? Or any other text editing behavior. This use-case-dependent behavior must have some "correct" behavior that is standardized somewhere? Like a way to query what should be treated like a single…

> Or typing 'o' + '¨' to produce 'ö' but realizing you wanted to type 'ô', there just one backspace press would revert it to 'o' again and you could press '^' to get the 'ô'.

This is a good example because in German I would expect 'o' + '¨' + to leave no character at all while in French I would expect 'e' + '`' + to leave the e behind because in my mind it was a typo.

The rendering of brahmic- and arabic-derived scripts makes these choices even more interesting.

Re: What every software developer must know about Unicode in 2023

#322

Post by gpvos »

superq wrote: post #191
ericmcer wrote: post #135

Earlier quoted context omitted.

I don't think he added moving cursors all over the page because he thought it was good UI/UX, he knows what he is doing.

This is seemingly self-contradictory. Perhaps you could explain your reasoning further?

Doing bad things is their idea of fun.

Re: What every software developer must know about Unicode in 2023

#323

Post by bluGill »

eviks wrote: post #124
tialaramex wrote: post #42

Earlier quoted context omitted.

The writing systems were already like this when we got them. Unicode's "total mess" mostly just reflects that. Of course it would be convenient for you, the programmer, if the users wanted the software to do whatever was easiest for you, but obviously they want what's easiest for them, not you.

How is it easiest "for them" to have the mess instead of having the newer standard be less messy?

because the current mess means all their old stuff still works. ASCII is good so long as you only need English (or any other latin languages without the various accents), which was good enough for a long time - and ASCII was also carefully designed to make programming easier - flip one bit changes lower/uppercase for example, but there are more things it makes easy. By the time we realized we actually care about the rest or the world it was too late to make a nice system.

Re: What every software developer must know about Unicode in 2023

#324

Post by arp242 »

WorldMaker wrote: post #297
arp242 wrote: post #278

Earlier quoted context omitted.

Well, in theory it's infinite, but in reality it's not of course. We've got 150K assigned codepoints assigned, leaving us with 950K unassigned codepoints. There's truly massive amounts of headroom. To be honest I think this argument is rather too abstract to be of any real use: if it's a theoretical problem that will never occur in reality then all I can say is: . But like I said: I'm not "against" combining marks, p…

The Unicode committees have addressed this for languages such as Latin, Cyrillic, and others and stated outright that decomposed forms should be preferred and decomposition canonical forms are generally the safest for interoperability and operations such as collation (sorting) and case folding (lowercase to uppercase transformations). Unicode can't get rid of the many precombined characters for a huge number of backw…

> The Unicode committees have addressed this for languages such as Latin, Cyrillic, and others and stated outright that decomposed forms should be preferred

Yes, and that only makes things worse since the overwhelming majority of documents (99.something% last time I checked) uses pre-composed. Also AFAIK just about everyone just ignores that recommendation.

This is a classic "reality should adjust to the standard" type of thinking. Previous comments about that: https://news.ycombinator.com/item?id=36984331

I suppose "e ZWJ %" is a bit better than Private Use as it will appear as "e%" if you don't have font support, but the fundamental problem of "won't work unless you spend effort" remains. For a specific niche (math, language study, something else) that's okay, but for "casual" usage: not so much. "Ship font with the document" like PDF and webfonts do is an option, but also has downsides and won't work in a lot of contexts, and still requires extra effort from the author.

I'm not saying it's completely impossible, but certainly harder than it used to be, arguably much harder. I could coin a new word right here and now (although my imagination is failing me to provide a humorous example at this moment) and if people like it, it will see usage. In 1960s HN when we would have exchanged these things over written letters, and it would have been trivial to propose a "e with % on top" too, but now we need to resort to clunky phrases like this (even for typewriters you can manually amend things, if you really wanted to).

Or let me put it this way: something like ‽ would see very little chance of being added to Unicode if it was coined today. Granted, it doesn't see that much use, but I do encounter it in the wild on occasion and some people like it (I personally don't actually, but I don't want to prevent other people from using it).

None of this is Unicode's fault by the way, or at least not directly – this is a generic limitation of computers.

Re: What every software developer must know about Unicode in 2023

#325

Post by gpvos »

coldpie wrote: post #168
toastercat wrote: post #119

Earlier quoted context omitted.

Anytime tonsky's site gets posted here, I'm reminded by how awful it is, which is ironic given his UI/UX background. The site's lightmode is a blinding saturated yellow, and if you switch into darkmode, it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.

Works like a normal website with JavaScript disabled. I didn't even know it did fancy junk until reading the comments here. NoScript saves the day again! I don't know how people can browse the web without it.

It is some time ago since I last used it, but I found that too many websites that I want to read require Javascript to even show you the main body of text, or a reasonable layout. Is that different now?

Re: What every software developer must know about Unicode in 2023

#326

Post by gumby »

mostlylurks wrote: post #291
School-Cotton wrote: post #231

Earlier quoted context omitted.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

The nordic layout(s) offer such a mechanism to allow people to type in letters that you'll find in various other European languages, even though the extra letters used in the languages themselves (ÅÄÖÆØ) are present as their own keys. Interestingly, the Swedish layout has no dedicated é key, although é occurs in some Swedish words.

In Swedish, Å, Ä, and Ö are actual letters of the alphabet, while é is used in foreign words. Like the English dieresis (e.g. in coöperate) is essentially unknown in the US and only occasionally used in England, so doesn't give rise to characters with dieresis on the keyboard.

Re: What every software developer must know about Unicode in 2023

#327

Post by hgs3 »

Dylan16807 wrote: post #314
hgs3 wrote: post #308

Earlier quoted context omitted.

> There are plenty of situations that warrant operating on code points Absolutely correct. All algorithms defined by the Unicode Standard and its technical reports operate on the code point. All 90+ character properties defined by the standard are queried for with the code point. The article omits this information and ironically links to the grapheme cluster break rules which operate on code points.

The article doesn't say not to use code points, it says you should not be iterating on them. Very rarely will you be implementing those algorithms. And if you're looking at character properties, the article says you should be looking at multiple together, which is correct.

> And if you're looking at character properties, the article says you should be looking at multiple together, which is correct.

I don't see where the article mentions Unicode character properties [1]. These properties are assigned to individual characters, not groups of characters or grapheme clusters.

> Very rarely will you be implementing those algorithms.

True, but character properties are frequently used, i.e. every time you parse text and call a character classification function like "isDigit" or "isControl" provided by your standard library you are in fact querying a Unicode character property.

[1] https://unicode.org/reports/tr44/#Properties

Re: What every software developer must know about Unicode in 2023

#328

Post by arp242 »

pests wrote: post #318
arp242 wrote: post #303

Earlier quoted context omitted.

> defaulting both skin tones to my settings But that's kind of wrong, no? The entire point is that you can choose both sides individually. What if you set it to black and want to kiss some white bloke? If anything that only underscores my point that it's too complex and that no one is using them (certainly not as intended anyway).

That's on Apple not on emojis. In the Windows 11 emoji picker it works like this: 1. Search "kissing". See two generic yellow people kissing. Notice a blue dot in the bottom right corner. 2. Clicking the emoji brings up previously used versions of the kissing emoji, with a + button. 3. Clicking + brings up a dialog like I described previously. Two generic figures at the top, then a row of skin tones. 4. You can click…

That seems like a lot of effort when you could have sent , , , , or any number of other emojis, which is what my point was.

Re: What every software developer must know about Unicode in 2023

#329

Post by posix86 »

gumby wrote: post #321
riggsdk wrote: post #270

Earlier quoted context omitted.

There are libraries that help with iterating both code-points and grapheme clusters... - but are there any of them that can help decide what to do for example when pressing backspace given an input string and a cursor position? Or any other text editing behavior. This use-case-dependent behavior must have some "correct" behavior that is standardized somewhere? Like a way to query what should be treated like a single…

> Or typing 'o' + '¨' to produce 'ö' but realizing you wanted to type 'ô', there just one backspace press would revert it to 'o' again and you could press '^' to get the 'ô'. This is a good example because in German I would expect 'o' + '¨' + to leave no character at all while in French I would expect 'e' + '`' + to leave the e behind because in my mind it was a typo. The rendering of brahmic- and arabic-derived scri…

But typing "ö" (e.g. swiss keyboard) and pressing delete & getting an o would be annoying af

Re: What every software developer must know about Unicode in 2023

#330

Post by pests »

arp242 wrote: post #328
pests wrote: post #318

Earlier quoted context omitted.

That's on Apple not on emojis. In the Windows 11 emoji picker it works like this: 1. Search "kissing". See two generic yellow people kissing. Notice a blue dot in the bottom right corner. 2. Clicking the emoji brings up previously used versions of the kissing emoji, with a + button. 3. Clicking + brings up a dialog like I described previously. Two generic figures at the top, then a row of skin tones. 4. You can click…

That seems like a lot of effort when you could have sent , , , , or any number of other emojis, which is what my point was.

You still can! People who want more customizations can do so too. Plus it only takes the initial setup per emoji at least.
Post reply on HN