Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

281–290 of 572 posts

Re: What every software developer must know about Unicode in 2023

#281

Post by TacticalCoder »

School-Cotton wrote: post #231
Findecanor wrote: post #217

Earlier quoted context omitted.

Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Nitpicking but most french keyboards have both ready-made keys for "é" and the few other commonly use keys and composing: hitting either '¨' or '^'. For example hitting '¨' then 'e' produces "ë".

Re: What every software developer must know about Unicode in 2023

#282

Post by WorldMaker »

qwerty456127 wrote: post #16

> The rest, about 800,000 code points, are not allocated at the moment. They could become characters in the future. Why is Tengwar still not in Uniclde officially? What's the problem with it?

The problem with Tengwar (and Klingon) is the problem with a lot of pop culture right now: copyright. The Tolkien Estate still exists and still litigiously upholds what it can of their copyright terms. CBS Viacom (Paramount) still claim a copyright interest in all the written forms of Klingon.

Copyright is not technically violated simply by encoding the characters into a plane such as one of Unicode's, that's an easy open and shut fair use, but Unicode principals have stated they don't want to pass on the copyright burden to font authors either, which would be sued if they tried to paint some of those characters. (Why encode something that fonts aren't allowed to produce?) That should also be fair use, but the law is complicated and copyright still so often today leans in favor of the Estates and major Corporations rather than fair use and the public commons.

(ETA: I'm hugely in favor that "conlang", constructed language, scripts such as these should be encoded by Unicode. I wish someday we fix the copyright problems of them.)

Re: What every software developer must know about Unicode in 2023

#285

Post by phforms »

Regarding UTF-8 encoding:

“And a couple of important consequences:

- You CAN’T determine the length of the string by counting bytes.

- You CAN’T randomly jump into the middle of the string and start reading.

- You CAN’T get a substring by cutting at arbitrary byte offsets. You might cut off part of the character.”

One of the things I had to get used to when learning the programming language Janet is that strings are just plain byte sequences, unaware of any encoding. So when I call `length` on a string of one character that is represented by 2 bytes in UTF-8 (e.g. `ä`), the function returns 2 instead of 1. Similar issues occur when trying to take a substring, as mentioned by the author.

As much as I love the approach Janet took here (it feels clean and simple and works well with their built-in PEGs), it is a bit annoying to work with outside of the ASCII range. Fortunately, there are libraries that can deal with this issue (e.g. https://github.com/andrewchambers/janet-utf8), but I wish they would support conversion to/from UTF-8 out of the box, since I generally like Janet very much.

One interesting thing I learned from the article is that the first byte can always be determined from its prefix. I always wondered how you would recognize/separate a unicode character in a Janet string since it may have 1-4 bytes length, but I guess this is the answer.

Re: What every software developer must know about Unicode in 2023

#286

Post by acqq »

pests wrote: post #259
tonsky wrote: post #185

Earlier quoted context omitted.

Hi, author here. In case you really want to know: no, it’s custom-made and works exactly as intended. There are two main reasons: 1. Fun. Modern internet is boring, most blog posts are just black text on white background. Hard to remember where you read what. And you can’t really have fun without breaking some expectations. 2. Sense of community. Internet is a lonely place, and I don’t necessarily like that. I like t…

I like your content but I do think you need to rethink #1. Fun is usless if no one wants to show up because they are annoyed.

Count me too to the group of "I was so distracted that I stopped reading."

Then the second thought was: I should again start to block js by default as much as I can.

Re: What every software developer must know about Unicode in 2023

#287

Post by mostlylurks »

orphea wrote: post #195
pif wrote: post #152

Earlier quoted context omitted.

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".

As a European, no I don't. á isn't used in my language, but my layout offers it via a dead-key-then-base-letter mechanism, and it is correctly treated as one unit when pressing backspace, anything else would feel incorrect. It would be even worse if such a thing happened for the letters that my layout offers individual buttons for (ÅÄÖ). Some languages do treat these as letters with attached modifiers, but many, including mine, treat them as indivisible letters that just happen to look similar to some others for historical reasons, and to treat them as combinations of base letters and diacritics would be completely incorrect, even if you typed them in using the dead-key-then-base-letter mechanism for some reason.

Re: What every software developer must know about Unicode in 2023

#288

Post by cryptonector »

> Another unfortunate example of locale dependence is the Unicode handling of dotless i in the Turkish language.

This isn't quite Unicode's fault, as the alternative would be to have two codepoints each for `i` and `I`, one pair for the Latin versions and one for the Turkish versions, and that would be very annoying too.

Whereas the Russian/Bulgarian situation is different. There used to be language tags in Unicode for that, but IIRC they got deprecated, and maybe they'll have to get undeprecated.

Re: What every software developer must know about Unicode in 2023

#289

Post by russellbeattie »

We need, desperately and without question, two Unicode symbols for bold and italic.

These are part of language and should not be an optional proprietary add on that can be skipped or deleted from text. We've been using the two "formats" to convey important information since the sixteenth century!!!

It boggles my mind that we can give flesh tone to emojis, yet not mark a word as bold or italic. It makes zero sense. Especially how easy it would be to implement. It would work exactly the same way: Letters following the mark would be formatted as bold or italic until a space character or equivalent.

Re: What every software developer must know about Unicode in 2023

#290

Post by layer8 »

pif wrote: post #152
jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

In terminals there is a distinction between single-width and double-width characters (east-asian characters, in particular). E.g. the three characters

    A美C
would take up the width of four ASCII monospace characters, the “美” being double-width.

Similarly, for composed characters like say the ligature “ff”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).

Post reply on HN