Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

301–310 of 572 posts

Re: What every software developer must know about Unicode in 2023

#301

Post by hgs3 »

jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

Everybody loves to debate what "character" means but nobody ever consults the standard. In the Unicode Standard a "character" is an abstract unit of textual data identified by a code point. The standard never refers to graphemes as "characters" but rather as user-perceived characters which the article omits.

Re: What every software developer must know about Unicode in 2023

#302

Post by titzer »

> The problem is, you don’t want to operate on code points. A code point is not a unit of writing; one code point is not always a single character. What you should be iterating on is called “extended grapheme clusters”, or graphemes for short.

It's best to avoid making overly-general claims like this. There are plenty of situations that warrant operating on code points, and it's likely that software trying and failing to make sense of grapheme clusters will result it in a worse screwup. Codepoints are probably the best default. For example, it probably makes the most sense for programming languages to define strings as arrays of code points, and not characters or 16-bit chunks or an encoding, or whatever.

Re: What every software developer must know about Unicode in 2023

#303

Post by arp242 »

pests wrote: post #279
arp242 wrote: post #245

Earlier quoted context omitted.

I mean, I don't dislike the concept personally. I actually really hate how HN strips them. But the technical implementation? Yeah, that could have gone a lot better IMHO. One must also wonder if some things really had to be added in the first place, e.g. for people kissing it's: (person|man|woman)(skin-tone)? ZWJ ZWJ ZWJ (person|man|woman)(skin-tone)? This is NOT a complaint about that they added diversity as such, i…

> there's 98 combinations to choose from here. Less than that since a default skin color can be set in most apps. I'm sure setting a gender will come soon so the entire first part of that emoji can be auto-guessed. Then its just showing the other options in the UI. Really all of this is UI design as even with the 98 combinations you can still display it as 4/5 options you drill down. > who finds it convenient to pick…

> defaulting both skin tones to my settings

But that's kind of wrong, no? The entire point is that you can choose both sides individually. What if you set it to black and want to kiss some white bloke?

If anything that only underscores my point that it's too complex and that no one is using them (certainly not as intended anyway).

Re: What every software developer must know about Unicode in 2023

#304

Post by Dylan16807 »

titzer wrote: post #302

> The problem is, you don’t want to operate on code points. A code point is not a unit of writing; one code point is not always a single character. What you should be iterating on is called “extended grapheme clusters”, or graphemes for short. It's best to avoid making overly-general claims like this. There are plenty of situations that warrant operating on code points, and it's likely that software trying and failin…

Situations such as?

Sometimes editing wants to go inside clusters but that's not code-point based either.

I'd say that in a big majority of situations, code that is indexing an array with code points is either treating the indexes as opaque pointers or is doing something wrong.

Re: What every software developer must know about Unicode in 2023

#305

Post by dizhn »

qwerty456127 wrote: post #23

> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to? Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11. Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last…

i İ

ı I

I symphatize with people who get this wrong. (I just saw some YouTube video have a title TÜRKIYE in a segment)

Even google keyboard can't seem to distinguish between I and İ. When I type "It", it suggests "İt's" which is quite pathetic.

Re: What every software developer must know about Unicode in 2023

#306

Post by Retr0id »

favorited wrote: post #295
thyselius wrote: post #202

Wonderful to learn more about Unicode. Does anyone know how to write a function (preferably in swift) to remove emoji? This is surprisingly hard (if the string can be any language, like English or Chinese). There’s been multiple attempts on Stackoverflow but they’re all missing some of them, as Unicode is so complex.

Here's a 1-liner, producing the string "text 0123 漢字": `String("text EMOJI 0123 漢字".unicodeScalars.filter({ !$0.properties.isEmojiPresentation }))` (I've had to substitute EMOJI for a smiley face, because HN is bad at text encoding.)

It's not a bug, HN deliberately strips emojis.

Re: What every software developer must know about Unicode in 2023

#307

Post by spacechild1 »

mplewis wrote: post #299
spacechild1 wrote: post #268

Earlier quoted context omitted.

It is obviously a joke (and a good one, I dare say). The fact that people seem to take it seriously says something about the contemporary state of webdesign :)

It would be a better joke if there were an option to turn the joke off. As it is, dark mode doesn't exist and the pointers occlude text.

> It would be a better joke if there were an option to turn the joke off.

As others have pointed out, reader mode works as expected.

Re: What every software developer must know about Unicode in 2023

#308

Post by hgs3 »

titzer wrote: post #302

> The problem is, you don’t want to operate on code points. A code point is not a unit of writing; one code point is not always a single character. What you should be iterating on is called “extended grapheme clusters”, or graphemes for short. It's best to avoid making overly-general claims like this. There are plenty of situations that warrant operating on code points, and it's likely that software trying and failin…

> There are plenty of situations that warrant operating on code points

Absolutely correct. All algorithms defined by the Unicode Standard and its technical reports operate on the code point. All 90+ character properties defined by the standard are queried for with the code point. The article omits this information and ironically links to the grapheme cluster break rules which operate on code points.

Re: What every software developer must know about Unicode in 2023

#309

Post by jcranmer »

riggsdk wrote: post #270
jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

There are libraries that help with iterating both code-points and grapheme clusters... - but are there any of them that can help decide what to do for example when pressing backspace given an input string and a cursor position? Or any other text editing behavior. This use-case-dependent behavior must have some "correct" behavior that is standardized somewhere? Like a way to query what should be treated like a single…

Some platforms (e.g., Android) have methods specifically for asking how to edit a string following a backspace. However, there's no standard Unicode algorithm to answer the question (and I strongly suspect that it's something that's actually locale-dependent to a degree).

On further reflection, probably the best starting point for string editing on backspace is to operate on codepoints, not grapheme clusters. For most written languages, the various elements that make up a character are likely to be separate codepoints. In Latin text, diacritics are generally precomposed (I mean, you can have a + diacritic as opposed to precomposed ä in theory, but the IME system is going to spit out ä anyways, even if dead keys are used). But if you have Indic characters or Hangul, the grapheme cluster algorithm is going to erroneously combine multiple characters into a single unit. The issue is that the biggest false positive for a codepoint-based algorithm is emoji, and if you're a monolingual speaker whose only exposure to complex written scripts is Unicode emoji, you're going to incorrectly generalize it for all written languages.

Re: What every software developer must know about Unicode in 2023

#310

Post by School-Cotton »

TacticalCoder wrote: post #281
School-Cotton wrote: post #231

Earlier quoted context omitted.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Nitpicking but most french keyboards have both ready-made keys for "é" and the few other commonly use keys and composing: hitting either '¨' or '^'. For example hitting '¨' then 'e' produces "ë".

You are right, thanks.
Post reply on HN