Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

311–320 of 572 posts

Re: What every software developer must know about Unicode in 2023

#311

Post by School-Cotton »

greenshackle2 wrote: post #252
School-Cotton wrote: post #231

Earlier quoted context omitted.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Which French layout would that be? I've never seen a French keyboard where this is true. French is my native language. On layouts I'm familiar with, some accented letters have separate keys like é, but not all, the others are made by composing an accent key with a letter.

You're right, sorry. I had forgotten about the ^ and ¨ keys.

Re: What every software developer must know about Unicode in 2023

#312

Post by wgjordan »

TacticalCoder wrote: post #292

> For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points! It's a poor and misleading example for it is definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there (french is the language where 'é' is the most common). 'é' is U+00F9, one codepoint, definitely not two. Now you could say: but it…

The author explains normalization in its own entire section several paragraphs later (Why is "Å" !== "Å" !== "Å"?).

Re: What every software developer must know about Unicode in 2023

#313

Post by phantom784 »

lucideer wrote: post #293
TRiG_Ireland wrote: post #262

Earlier quoted context omitted.

> US English spellings & conventions are quite a departure from other dialects. As far as the written, formal language is concerned, English really has only three dialects: US American, Canadian, and everywhere else. There are some other subtle differences (such as "robots" for traffic lights in South Africa, or "minerals" for fizzy drinks in Ireland¹), but that's pretty much it. ¹ Yes, this isn't just slang in Irela…

> As far as the written, formal language is concerned, English really has only three dialects This is true, but I don't see why the "formal" qualifier is needed here :) There are much more than 3 dialects of English, both written & spoken. Especially there's a fair few extremely common notable differences in (casual, written) Irish English: the word "amn't" (among other less common contractions), the alternative pres…

I guess it's a question as to how many varieties of spelling you want to make available as "translations" in software (e.g. color vs colour, tire vs tyre).

There's plenty of regional variants just within the US, but "en_us" covers the whole country.

Re: What every software developer must know about Unicode in 2023

#314

Post by Dylan16807 »

hgs3 wrote: post #308
titzer wrote: post #302

> The problem is, you don’t want to operate on code points. A code point is not a unit of writing; one code point is not always a single character. What you should be iterating on is called “extended grapheme clusters”, or graphemes for short. It's best to avoid making overly-general claims like this. There are plenty of situations that warrant operating on code points, and it's likely that software trying and failin…

> There are plenty of situations that warrant operating on code points Absolutely correct. All algorithms defined by the Unicode Standard and its technical reports operate on the code point. All 90+ character properties defined by the standard are queried for with the code point. The article omits this information and ironically links to the grapheme cluster break rules which operate on code points.

The article doesn't say not to use code points, it says you should not be iterating on them.

Very rarely will you be implementing those algorithms. And if you're looking at character properties, the article says you should be looking at multiple together, which is correct.

Re: What every software developer must know about Unicode in 2023

#315

Post by jeddy3 »

uxp8u61q wrote: post #189
pif wrote: post #162

> The minimum every software developer must know about Unicode Just a nitpick... Once more, as it is typical on HN, web programming is confused with the entire universe of software development. There are plenty of software realms where ASCII not only is enough, but it actually MUST be enough.

This kind of assertiveness leads to garbage like C++ still not supporting UTF8 properly in 2023. My name contains diacritics. I am so, so, so tired of trying to work around information systems - not just web frontends - designed by people who don't care or worse, don't want to care. "Web" programmers can care all they want about Unicode, but if the backend people didn't deal properly with text encoding, then somethin…

I can name one. At my job we do the kind of embedded programming were encoders inside machines send data to each other. Like reading optical sensors and sending bits indicating state to other controllers.

We absolutely do not "need" to know about Unicode, outside of interest about other realms.

Re: What every software developer must know about Unicode in 2023

#316

Post by jcranmer »

bajsejohannes wrote: post #28
rurban wrote: post #17

The Why is "Å" !== "Å" !== "Å"? section still strikes me as wrong. The strings are equal even when the representations differ.

I just not sure why they put in the "Angstrom symbol" to begin with. If you do, then why isn't the "meter symbol" (m) also represented? Fortunately, it seems like it's marked as deprecated: https://en.wikipedia.org/wiki/Angstrom#Symbol

> I just not sure why they put in the "Angstrom symbol" to begin with.

Frequently, the answer to this is "some obscure character set had this as a distinct symbol." In this case, blame the Japanese: https://en.wikipedia.org/wiki/JIS_X_0208

Which is why there's an 'mm' and 'cm' and other random symbols: https://www.compart.com/en/unicode/block/U+3300

Re: What every software developer must know about Unicode in 2023

#317

Post by bluGill »

OfSanguineFire wrote: post #201
BoxOfRain wrote: post #139

Earlier quoted context omitted.

Yeah I'd rather not have my British English dialect seen as second-class in a world of American English ideally which is what having a red document full of 'errors' implies in those sorts of situations. It's sometimes not a trivial distinction either, for example I've heard of cases where surprised British redditors have found themselves banned from American subreddits for being homophobic when they were actually tal…

I would think a lot of mods, who are either Highly Online Americans or their weirdo equivalents in other countries, are well aware of the UK usage, but simply expect Brits to give it up in order to avoid offending Americans and the global Reddit community that largely takes American-style sensitivity as its orthodoxy. And considering that Reddit corporate feels that anything that could stir up such outrage is bad for…

It doesn't matter if you use UK or US spelling you are wrong. I wish we would adopt the international phonetic alphabet I might have a chance of spelling things correctly.

Re: What every software developer must know about Unicode in 2023

#318

Post by pests »

arp242 wrote: post #303
pests wrote: post #279

Earlier quoted context omitted.

> there's 98 combinations to choose from here. Less than that since a default skin color can be set in most apps. I'm sure setting a gender will come soon so the entire first part of that emoji can be auto-guessed. Then its just showing the other options in the UI. Really all of this is UI design as even with the 98 combinations you can still display it as 4/5 options you drill down. > who finds it convenient to pick…

> defaulting both skin tones to my settings But that's kind of wrong, no? The entire point is that you can choose both sides individually. What if you set it to black and want to kiss some white bloke? If anything that only underscores my point that it's too complex and that no one is using them (certainly not as intended anyway).

That's on Apple not on emojis.

In the Windows 11 emoji picker it works like this:

1. Search "kissing". See two generic yellow people kissing. Notice a blue dot in the bottom right corner.

2. Clicking the emoji brings up previously used versions of the kissing emoji, with a + button.

3. Clicking + brings up a dialog like I described previously. Two generic figures at the top, then a row of skin tones.

4. You can click on each generic person and choose a gender, then select a skin tone. You can do this for each person in the group.

5. Click done. This emoji is now in your default emoji list and you won't need to recreate it again.

Re: What every software developer must know about Unicode in 2023

#320

Post by thyselius »

favorited wrote: post #295
thyselius wrote: post #202

Wonderful to learn more about Unicode. Does anyone know how to write a function (preferably in swift) to remove emoji? This is surprisingly hard (if the string can be any language, like English or Chinese). There’s been multiple attempts on Stackoverflow but they’re all missing some of them, as Unicode is so complex.

Here's a 1-liner, producing the string "text 0123 漢字": `String("text EMOJI 0123 漢字".unicodeScalars.filter({ !$0.properties.isEmojiPresentation }))` (I've had to substitute EMOJI for a smiley face, because HN is bad at text encoding.)

Thanks. Unfortunately both .isEmojiPresentation && .isEmoji leaves many emojis out, like red heart and many other.
Post reply on HN