Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

21–30 of 572 posts

Re: What every software developer must know about Unicode in 2023

#21
post #6

https://tonsky.me/blog/unicode/overview@2x.png Wow, what abominable mix of decimal and hexadecimal.

Where are the decimal numbers in that image?

It goes 90000..9FFFF then 100000..10FFFF. The latter should have been A0000..AFFFF.

So the author is using hex for the last four digits and decimal for the remaining ones.

Re: What every software developer must know about Unicode in 2023

#22

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

turned off javascript as soon as I saw it. Like trying to read with twenty mosquitos in your face.

Re: What every software developer must know about Unicode in 2023

#23
> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to?

Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11.

Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last time I checked), I have always been setting the locale to En-US even though I have never been to America. This leads to a number of annoyances though. E.g. LibreOffice always creates new documents for the Letter paper format and I have to switch it to A4 manually every time. It's even worse on Linux where locales appear to be less easy to customize than in Windows. Windows always offered a handy configuration dialog to granularly tweak your locale choosing what measures system you prefer, whether your weeks begin on sundays or mondays and even define your preferred date-time format templates fully manually.

A less-spoken about problem is Windows' system-wide setting for the default legacy codepage. I happen to use single-language legacy (non-Unicode) apps made by people from a number of very different countries. Some apps (e.g. I can remeber the Intel UHD Windows driver config app) even use this setting (ignoring the system locale and system UI language) to detect your language and render their whole UI in it.

> English (USA), English (UK)

This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. To us presence of a huge number of these (don't forget en-AU, en-TT, en-ZW etc - there are more!) in the options lists brings only annoyance, especially when one chooses some non-US one and this opens another can of worms.

By the way I wonder how do string capitalization and comparision functions manage to work on computers of people who use both English and Turkish actively (Turkish locale distinguishes between dotted and undotted İ).

Re: What every software developer must know about Unicode in 2023

#24
post #17

The Why is "Å" !== "Å" !== "Å"? section still strikes me as wrong. The strings are equal even when the representations differ.

They are logically equal (that is, they represent the same text in an abstract way), but computing this equality in practice is expensive, because you first need to normalize the strings then compare.

Most languages, when comparing strings, skip the normalization and just compare string bytes as is (or, if the string is interned, compare just the pointer)

Re: What every software developer must know about Unicode in 2023

#26
post #12

>That gives us a space of about 11 million code points. About 170,000, or 15%, are currently defined. An additional 11% are reserved for private use. The rest, about 800,000 code points, are not allocated at the moment. They could become characters in the future. 1.1 million?

Yeah, the author's numbers are off by a "0". It should be "1,700,000" and "8,000,000".

Re: What every software developer must know about Unicode in 2023

#27
Unicode is a total mess. In a sane system, "extended grapheme clusters" would equal "codepoints" and it wouldn't make a difference for 99% of languages. Now we ended up with grapheme clusters, normalization, decomposition, composition, Zalgo text, etc. But instead of deprecating this nonsense, Unicode doubled down with composed Emojis.

Re: What every software developer must know about Unicode in 2023

#28
post #17

The Why is "Å" !== "Å" !== "Å"? section still strikes me as wrong. The strings are equal even when the representations differ.

I just not sure why they put in the "Angstrom symbol" to begin with. If you do, then why isn't the "meter symbol" (m) also represented?

Fortunately, it seems like it's marked as deprecated: https://en.wikipedia.org/wiki/Angstrom#Symbol

Re: What every software developer must know about Unicode in 2023

#30

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

yeah....why on earth would someone want their webpage to do this, especially if they have text they'd presumably want you to read?
Post reply on HN