Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

131–140 of 572 posts

Re: What every software developer must know about Unicode in 2023

#131

Post by masklinn »

danbruc wrote: post #114
bumbledraven wrote: post #109

> what to you think "ẇ͓̞͒͟͡ǫ̠̠̉̏͠͡ͅr̬̺͚̍͛̔͒͢d̠͎̗̳͇͆̋̊͂͐".length should be? This is a nice example of the kind of thing we need to think about when defining a measure of length for Unicode strings.

Four. Obviously. The more interesting question is whether the Unicode rules actually give that answer. EDIT: Just checked it using the first online tool [1] that came up and it indeed says four. So all is good. [1] https://onlinetools.com/unicode/extract-unicode-graphemes

It should be 4 as long as you count the grapheme clusters which is what e.g. Swift does (hence String#count being O(n)).

In Javascript, you can get the same information through Intl.Segmenter, segments by grapheme cluster by default.

Re: What every software developer must know about Unicode in 2023

#132

Post by hot_gril »

ilyt wrote: post #19
davidham wrote: post #10

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

I see nice crisp black text on white background because apparently server melted down

I saw that, except half the images weren't loading, and there was just one mouse pointer.

Re: What every software developer must know about Unicode in 2023

#133

Post by WillAdams »

Just had this come up at work --- needed a checkbox in Microsoft Word --- oddly the solution to entering it was to use the numeric keypad, hold down the alt key and then type out 128504 which yielded a check mark when the Arial font was selected _and_ unlike Insert Symbol and other techniques didn't change the font to Segoe UI Symbol or some other font with that symbol.

Oddly, even though the Word UI indicated it was Arial, exporting to a PDF and inspecting that revealed that Segoe UI Symbol was being used.

As I've noted in the past, "If typography was easy, Microsoft Word wouldn't be the foetid mess which it is."

Re: What every software developer must know about Unicode in 2023

#135

Post by ericmcer »

toastercat wrote: post #119
davidham wrote: post #10

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

Anytime tonsky's site gets posted here, I'm reminded by how awful it is, which is ironic given his UI/UX background. The site's lightmode is a blinding saturated yellow, and if you switch into darkmode, it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.

I don't think he added moving cursors all over the page because he thought it was good UI/UX, he knows what he is doing.

Re: What every software developer must know about Unicode in 2023

#136

Post by swores »

gwervc wrote: post #118
gumby wrote: post #64

This is quite a good write up. An answer to one of the author's questions: > Why does the fi ligature even have its own code point? No idea. On of the principles of Unicode is round trip compatibility. That is you should be able to read in a file encoded with some obsolete coding system and write it out again properly. Maybe frob it a bit with your unicode-based tools first. This is a good principle, though less usefu…

The circled digits as code points are very nice to have precisely because they are available in applications that don't support them otherwise... which is actually most of the software I can think of (Notepad, Apple Notes, chat applications, most websites, etc).

Can you write them with iOS keyboard? Or when you say Apple Notes and chat apps you just mean from desktop?

Edit ①: seems the answer is not with the default iOS keyboard, but possible to paste it and perhaps possible with a third party keyboard that I'm not keen on trying (unless I hear of a keyboard that's both genuinely useful / better than default, and that doesn't send keystrokes to the developer - though I can't remember if the latter is even a risk on iOS, better go search about that next..)

Re: What every software developer must know about Unicode in 2023

#137

Post by eviks »

nwellnhof wrote: post #27

Unicode is a total mess. In a sane system, "extended grapheme clusters" would equal "codepoints" and it wouldn't make a difference for 99% of languages. Now we ended up with grapheme clusters, normalization, decomposition, composition, Zalgo text, etc. But instead of deprecating this nonsense, Unicode doubled down with composed Emojis.

But precomposing all the potential combinations is less sane than the current mess (and you can outlaw Zalgo in the standard if you think it's a serious issue)

Also, the % should measure people, not languages, that would greatly decrease the imaginary 99%

Re: What every software developer must know about Unicode in 2023

#138

Post by masklinn »

swores wrote: post #136
gwervc wrote: post #118

Earlier quoted context omitted.

The circled digits as code points are very nice to have precisely because they are available in applications that don't support them otherwise... which is actually most of the software I can think of (Notepad, Apple Notes, chat applications, most websites, etc).

Can you write them with iOS keyboard? Or when you say Apple Notes and chat apps you just mean from desktop? Edit ①: seems the answer is not with the default iOS keyboard, but possible to paste it and perhaps possible with a third party keyboard that I'm not keen on trying (unless I hear of a keyboard that's both genuinely useful / better than default, and that doesn't send keystrokes to the developer - though I can't…

You can copy/paste them from a character board, a dedicated website, or even the wiki.

Re: What every software developer must know about Unicode in 2023

#139

Post by BoxOfRain »

DoughnutHole wrote: post #105
qwerty456127 wrote: post #23

> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to? Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11. Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last…

> I doubt many English speakers care to distinguish between English dialects It's worthwhile purely for the sake of autocorrect/typo highlighting in text-editing software. I don't miss the days of spelling a word correctly in my version of English but still being stuck with the visual noise of red highlighting up and down the document because it doesn't conform to US English.

Yeah I'd rather not have my British English dialect seen as second-class in a world of American English ideally which is what having a red document full of 'errors' implies in those sorts of situations.

It's sometimes not a trivial distinction either, for example I've heard of cases where surprised British redditors have found themselves banned from American subreddits for being homophobic when they were actually talking innocently enough about cigarettes!

Re: What every software developer must know about Unicode in 2023

#140

Post by l72 »

lucideer wrote: post #129
qwerty456127 wrote: post #23

> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to? Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11. Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last…

As an Irish person, while we have en_IE which is great (and solves most of the problems you list re: Euro-centric defaults + English), I'd still quite like to have an even more broad / trans-language / "cosmopolitan" locale to use. I mainly type in English but occasionally other languages - I use a combination of Mac & Linux - macOS has an (off-by-default but enable-able) lang-changer icon in the tray that is handy e…

I write in multiple languages daily on Linux, including English, Russian, and Chinese. Switching keyboards (at least with gnome) is a simple super-space.

While in my default (English) layout, it is easy enough to add in accents other characters using the compose key (right alt). So right-alt+'+a = á or right-alt+"+u = ü. I much prefer this over the long press as I can do it quickly and seamlessly without having to wait on feedback. Granted, it is not as discoverable, but once you are comfortable, it in my opinion is a better system.

Post reply on HN