Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

461–470 of 572 posts

Re: What every software developer must know about Unicode in 2023

#461

Earlier quoted context omitted.

You are right, thanks.

The AZERTY layout is nothing if not inconsistent.

For this specific example, it is actually quite pragmatic. "é" being used many orders of magnitude more often than "ë" in French, it makes sense for it to have its own key.

Re: What every software developer must know about Unicode in 2023

#463
post #300

Earlier quoted context omitted.

That's sounds a bit false to me. The Umlaute (ä,ö, ü) and the "eszett" ß are actually part of the German alphabet[1]. Also it is kinda weird to describe them as ligatures of the original letters and the diaeresis, because while this is what they started out as a long time ago, they are just their own letters now (as opposed to "real" stylistic ligatures like combining fi into one glyph). The advice your kid was told…

They sure are letters, but they aren't generally thought of as being in the alphabet (which seems to be why they are just kinda tacked on after a space on wikipedia) and get ordered as if they where just the base letter (mostly)

Note that in Swedish they are considered letters, and in Danish and Norwegian Æ, Ø and Å are letters.

Re: What every software developer must know about Unicode in 2023

#464

Earlier quoted context omitted.

I don't think he added moving cursors all over the page because he thought it was good UI/UX, he knows what he is doing.

I'm having a hard time reconciling "he knows what he is doing" with him making his site practically unusable without a reader mode, which by the way, not every browser supports (especially on mobile).

Don't even think of switching on the dark (night) mode with that attitude! :D

I really enjoyed the tongue in cheek design. I think every modern browser either allows you to turn on reader mode (especially on mobile) or just turn off CSS. This particular article works excellently even in w3m.

Re: What every software developer must know about Unicode in 2023

#465

Earlier quoted context omitted.

Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Press the key to the left of 1 (not the numpad) or the right of the Eszett (sharp S) on the German QWERTZ keyboard and you probably hit a dead key. There are dedicated keys for the German umlauts and Eszett but these are for French accents in loan words: â, á and à, e.g. as in Café.

It's worth mentioning of course that there are no-dead-keys variants of the keyboard layout but this has been pretty much the norm on Windows since the 1990s I think.

Re: What every software developer must know about Unicode in 2023

#466
post #64

This is quite a good write up. An answer to one of the author's questions: > Why does the fi ligature even have its own code point? No idea. On of the principles of Unicode is round trip compatibility. That is you should be able to read in a file encoded with some obsolete coding system and write it out again properly. Maybe frob it a bit with your unicode-based tools first. This is a good principle, though less usefu…

> That's also why things like digits with a circle around them exist: they were in some old Japanese character set.

Replace "digits with a circle around them" with "emojis" and that's also true.

Re: What every software developer must know about Unicode in 2023

#467
post #272

> The only modern language that gets it right is Swift: print("...".count) // => 1 And Erlang/Elixir! I guess they are not "cool" enough. But they correctly interpret that as one grapheme cluster. % erl +pc unicode > string:length("..."). 1 (... here is the U+1F926 U+1F3FB U+200D U+2642 U+FE0F emoji)

The author does refer to Elixir further down:

> UPD: Erlang/Elixir seem to be doing the right thing, too.

Re: What every software developer must know about Unicode in 2023

#468

Earlier quoted context omitted.

I'm not Korean but seeing that said of the Hangul example definitely made me pause - I doubt Koreans think of that example as a single grapheme (open to correction), though it is an excellent example all the same since it demonstrates the complexity of defining "units" consistently across language. It reminds me a little of Open Street Map's inconsistent administrative hierarchies ("states", "countries", "counties",…

As a native Korean, I can confirm that "각" is perceived as a single character. But the example itself is bad anyway because everyone use a precomposed form U+AC01 instead of U+1100 U+1161 U+11A8 instead (they are canonically equivalent). This is more clear when you also consider a compatibility representation "ㄱㅏㄱ" U+3131 U+314F U+3131, which is same to "각" after compatibility normalizations (NFKC or NFKD), but perce…

Thanks for the clarification, that's interesting.

My impression before was it would be considered a single "entity" (not the same as a roman-alphabetic word, but not a character either) containing 3 characters.

Re: What every software developer must know about Unicode in 2023

#469
post #64

This is quite a good write up. An answer to one of the author's questions: > Why does the fi ligature even have its own code point? No idea. On of the principles of Unicode is round trip compatibility. That is you should be able to read in a file encoded with some obsolete coding system and write it out again properly. Maybe frob it a bit with your unicode-based tools first. This is a good principle, though less usefu…

> they were in some old Japanese character set This implies that they're obsolete, but they're not -- they're still in very common use today. You can type them in Japanese by typing まる (maru, circle) and the number, then pick it out of the IME menu. Some IMEs will bring them up if you just type the number and go to the menu, too. :)

What do the Japanese use the circled numbers for?

Re: What every software developer must know about Unicode in 2023

#470

Earlier quoted context omitted.

> As far as the written, formal language is concerned, English really has only three dialects This is true, but I don't see why the "formal" qualifier is needed here :) There are much more than 3 dialects of English, both written & spoken. Especially there's a fair few extremely common notable differences in (casual, written) Irish English: the word "amn't" (among other less common contractions), the alternative pres…

I guess it's a question as to how many varieties of spelling you want to make available as "translations" in software (e.g. color vs colour, tire vs tyre). There's plenty of regional variants just within the US, but "en_us" covers the whole country.

That's a fair point - even in tiny tiny Ireland there's many regional dialects, with larger countries there'll typically be far more.

I guess the simple answer to that is: how much interest is there in maintenance. I don't think there's any compelling reason not to create something: if there's insufficient interest in maintenance that's an imperfect but reasonable proxy for utility.

I'm not aware of any maintained en_US_Xyz languages but it might be pretty cool if someone started. There's precedence in a few other languages, like no_NO_NY, zh_Hans_HK, etc.

Post reply on HN