Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

141–150 of 572 posts

Re: What every software developer must know about Unicode in 2023

#141

Post by nabla9 »

>3 Grapheme Cluster Boundaries

>It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a user-perceived character: users think of it as a single character, yet is actually represented by two Unicode code points. These user-perceived characters are approximated by what is called a grapheme cluster, which can be determined programmatically.

Re: What every software developer must know about Unicode in 2023

#142

Post by jcranmer »

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong.

The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." Like so many other things in Unicode, the correct answer is use-case dependent.

(And for this reason, String iteration should be based on codepoints--it's the fundamental level on which Unicode works, and whatever algorithm you want to use to derive the correct answer for your purpose will be based on codepoint iteration. hsivonen's article (https://hsivonen.fi/string-length/), linked in this one, does try to explain why extended grapheme clusters is the wrong primitive to use in a language.)

Re: What every software developer must know about Unicode in 2023

#143

Post by hot_gril »

dathinab wrote: post #46

> The only modern language that gets it right is Swift: I disagree. What is the "right" things is use-case dependent. For UI it's glyph bases, kinda, more precise some good enough abstraction over render width. For which glyphs are not always good enough but also the best you can get without adding a ton of complexity. But for pretty much every other use-case you want storage byte size. I mean in the UI you care abou…

Swift made an effort to handle grapheme clusters but severely over-complicated strings by exposing performance details to users. Look at the complex SO answers to what should be simple questions, like finding a substring: https://news.ycombinator.com/item?id=32325511 , many of which changed several times between Swift versions

I was working on an app in Swift that needed full emoji support once. Team ended up writing our own string lib that stores things as an array of single-character Swift strings.

Re: What every software developer must know about Unicode in 2023

#144

Post by gwbas1c »

moelf wrote: post #128

> The only modern language that gets it right is Swift: arguably not true: julia> using Unicode # for some reason HN doesn't allow emoji julia> graphemes(" ") length-1 GraphemeIterator{String} for " " help?> graphemes search: graphemes graphemes(s::AbstractString) -> GraphemeIterator Return an iterator over substrings of s that correspond to the extended graphemes in the string, as defined by Unicode UAX #29. (Roughl…

Julia is not a major language like Swift.

Re: What every software developer must know about Unicode in 2023

#145

Post by sebstefan »

hwillis wrote: post #22
davidham wrote: post #10

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

turned off javascript as soon as I saw it. Like trying to read with twenty mosquitos in your face.

hey be nice to my mouse cursor

Re: What every software developer must know about Unicode in 2023

#146

Post by Nevermark »

permo-w wrote: post #13
bqmjjx0kac wrote: post #7

Earlier quoted context omitted.

The mustard background with black text is harsh on the eyes.

strange. I quite like it

Me too. I get the impression of a very saturated off-white yellow.

But any more saturation and it would go all mustard-electric on me.

That's an interesting observation on variation of saturation response. Feels like useful knowledge for ... web site designers. Or any color crafter.

Re: What every software developer must know about Unicode in 2023

#147

Post by sdrothrock »

gumby wrote: post #64

This is quite a good write up. An answer to one of the author's questions: > Why does the fi ligature even have its own code point? No idea. On of the principles of Unicode is round trip compatibility. That is you should be able to read in a file encoded with some obsolete coding system and write it out again properly. Maybe frob it a bit with your unicode-based tools first. This is a good principle, though less usefu…

> they were in some old Japanese character set

This implies that they're obsolete, but they're not -- they're still in very common use today. You can type them in Japanese by typing まる (maru, circle) and the number, then pick it out of the IME menu. Some IMEs will bring them up if you just type the number and go to the menu, too. :)

Re: What every software developer must know about Unicode in 2023

#148

Post by eviks »

ebiester wrote: post #98
m3kw9 wrote: post #89

Unicode looks like a big over engineered standard that had 50 hands trying to put their mark in

It looks like that because Unicode is trying to solve a problem that everyone thinks is easy until they uncover the true extent of encoding human languages.

How does this explain surrogate pairs?

Re: What every software developer must know about Unicode in 2023

#149

Post by hot_gril »

toast0 wrote: post #126
dathinab wrote: post #72

Earlier quoted context omitted.

that's not the issue defaults matter as in they should things you can just use by-default without thinking about it as swift is deeply rooted in UI design having a default of glyphs make sense and as rust is deeply rooted in unix server and system programming utf-8 bytes make a lot of sense through the moment your language becomes more general purpose you could argue having a default in any way is wrong and it should…

> as in they should things you can just use by-default without thinking about it That time has passed. If you want to know the length of a string, you really should indicate what length type you mean.

There was no string.length in Swift for a while. Then they added one that just does what the user expects, get the number of grapheme clusters. If a user figures out that this isn't what they want, they can go use the other length method.

Re: What every software developer must know about Unicode in 2023

#150

Post by sebstefan »

nabla9 wrote: post #141

>3 Grapheme Cluster Boundaries >It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a use…

Oh my god, is there ever anything simple about unicode
Post reply on HN