Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

331–340 of 572 posts

Re: What every software developer must know about Unicode in 2023

#331

Post by gpvos »

tonsky wrote: post #185
dathinab wrote: post #60

Earlier quoted context omitted.

(sarcasm) It's revenge against anyone with certain kinds of visual impairments and/or concentration issues because the ex-spouse of the author which turned out to be a terrible person had such. (sarcasm try 2) It's revenge against anyone using JS on the net with the author trying to subtle hint that JS is bad. (realistic) It's probably on of: - the website is a static view of some collaborative tool which has that fu…

Hi, author here. In case you really want to know: no, it’s custom-made and works exactly as intended. There are two main reasons: 1. Fun. Modern internet is boring, most blog posts are just black text on white background. Hard to remember where you read what. And you can’t really have fun without breaking some expectations. 2. Sense of community. Internet is a lonely place, and I don’t necessarily like that. I like t…

2. I only understood that it was actual other people's mouse cursors when I read that here. So it didn't really engender a sense of community, although after some time I did think you are very good at modelling actual human mouse movements. Now that I know it, it's pretty neat though.

Re: What every software developer must know about Unicode in 2023

#333

Post by loeg »

Pretty clearly, "every software developer" doesn't need to understand Unicode with this level of familiarity, much like "every programmer" doesn't need to know the full contents of the 114 page Drepper paper. For example, I work on a GUID-addressed object store. Everything is in term of bytes and 128-bit UUIDs. Unicode is irrelevant to everyone on my team, and most adjacent teams. There is lots of software like this.

Re: What every software developer must know about Unicode in 2023

#334

Post by skitter »

samatman wrote: post #84

Please don't refer to codepoints as characters. Some are, some are not, it isn't a useful or informative approximation, it's just wrong. Unicode is a table which assigns unique numbers to different codepoints , most of which are characters. ZWJ is not a character at all, and extended grapheme clusters made of several codepoints are.

'Character' doesn't have a single meaning. ZWJ is a character according to definitions (2) and (3) in https://unicode.org/glossary/#character

Re: What every software developer must know about Unicode in 2023

#335

Post by jrockway »

hot_gril wrote: post #167
jetbalsa wrote: post #38

Earlier quoted context omitted.

I feel its the same as with any long standing computer system we have today. It was designed as more and more of the world came online and all the growing pains it came with. Could it be built from scratch today better? Yes. Will it? No. I suspect it will be around long after we are all dead. Same with IPv4 :V

Honestly I like ipv4 better than v6. I like having a NAT and easy addresses like 192.168.1.3 instead of fe80::210:5aff:feaa:20a2. They didn't need to mess with those things just to expand the address space, like how utf8 didn't require remapping ASCII.

IPv4.1 should have just had 39 bits, to be written like 999.999.999.999. (I know this wouldn't have actually had much effect, nobody is going to add new routes in the middle of "class A" spaces that already existed, so it would just give those that already had IP addresses more IP addresses. Additionally, people really abuse decimal addresses in horrifying ways; for example, Fios steals 192.168.1.100-192.168.1.150 for its TV service, and that range doesn't really correspond to anything that you can mask off in binary. It only makes sense in decimal, which is not what any underlying machinery uses. They should have given themselves a /26 or something. You get 3 for yourself (modulo the broadcast and gateway address), and they get 1 for TV.)

Re: What every software developer must know about Unicode in 2023

#336

Post by spacechild1 »

TacticalCoder wrote: post #292

> For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points! It's a poor and misleading example for it is definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there (french is the language where 'é' is the most common). 'é' is U+00F9, one codepoint, definitely not two. Now you could say: but it…

Next time read the whole article before accusing the author of incompetence!

However, the author could have added a small note, e.g. "(Unicode normalization will be convered in a later section.)", to prevent knowledgable readers from rage quitting :)

Re: What every software developer must know about Unicode in 2023

#337

Post by kazinator »

If you have to recognize a grapheme cluster, it will be easier to do that from a sequence of code points, than from UTF-8.

It's like saying that we don't need to tokenize, because you never want to deal with tokens anyway, but phrase structures!

Mmkay, whatever ...

Re: What every software developer must know about Unicode in 2023

#338

Post by Reefersleep »

School-Cotton wrote: post #231
Findecanor wrote: post #217

Earlier quoted context omitted.

Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Danish is one.

Re: What every software developer must know about Unicode in 2023

#339

Post by TRiG_Ireland »

lucideer wrote: post #293
TRiG_Ireland wrote: post #262

Earlier quoted context omitted.

> US English spellings & conventions are quite a departure from other dialects. As far as the written, formal language is concerned, English really has only three dialects: US American, Canadian, and everywhere else. There are some other subtle differences (such as "robots" for traffic lights in South Africa, or "minerals" for fizzy drinks in Ireland¹), but that's pretty much it. ¹ Yes, this isn't just slang in Irela…

> As far as the written, formal language is concerned, English really has only three dialects This is true, but I don't see why the "formal" qualifier is needed here :) There are much more than 3 dialects of English, both written & spoken. Especially there's a fair few extremely common notable differences in (casual, written) Irish English: the word "amn't" (among other less common contractions), the alternative pres…

Well, quite. If we include any one or more of the following three categories — formal spoken language, informal spoken language, informal written language — then there's definitely far more than three dialects of English. But formal spoken language really has only the three.

Re: What every software developer must know about Unicode in 2023

#340

Post by JohnFen »

lelanthran wrote: post #256
uxp8u61q wrote: post #242

Earlier quoted context omitted.

What do you mean by "work"? That you can store arbitrary bytes in a string? That's a pretty low bar.

> What do you mean by "work"? That you can store arbitrary bytes in a string? That's a pretty low bar. That's all that's needed for a backend language. The backend does not need to understand, or even acknowledge the existence, of grapheme clusters. Because the frontend is already having to understand all of this, it should be normalising any multi-codepoint ambiguous cluster anyway.

The backend never needs to do things like figure out how long a string is or search for one string in a database of other strings?
Post reply on HN