What every software developer must know about Unicode in 2023 • Hacker News Board

Re: What every software developer must know about Unicode in 2023

#351

Post by thefringthing » Mon, Oct 02, 2023, 9:33 PM UTC

> Unicode is a standard that aims to unify all human languages, both past and present, and make them work with computers.

This is doubly wrong.

First, it conflates languages and writing systems. Malay and English use the same writing system but are different languages. American Sign Language is a language, but it has no standard or widely-adopted writing system. Hakka is a language, but Hakka speakers normally write in Modern Standard Mandarin, a different language.

Second, it's not that case that Unicode aims to encode all writing systems. For example, there are many hobbyist neographies (constructed writing systems) which will not be included in Unicode.

Re: What every software developer must know about Unicode in 2023

#352

Post by [deleted] » Mon, Oct 02, 2023, 9:35 PM UTC

jrockway wrote: post #335

hot_gril wrote: post #167

Earlier quoted context omitted.

Honestly I like ipv4 better than v6. I like having a NAT and easy addresses like 192.168.1.3 instead of fe80::210:5aff:feaa:20a2. They didn't need to mess with those things just to expand the address space, like how utf8 didn't require remapping ASCII.

IPv4.1 should have just had 39 bits, to be written like 999.999.999.999. (I know this wouldn't have actually had much effect, nobody is going to add new routes in the middle of "class A" spaces that already existed, so it would just give those that already had IP addresses more IP addresses. Additionally, people really abuse decimal addresses in horrifying ways; for example, Fios steals 192.168.1.100-192.168.1.150 fo…

[deleted]

Re: What every software developer must know about Unicode in 2023

#353

Post by gumby » Mon, Oct 02, 2023, 9:35 PM UTC

posix86 wrote: post #329

gumby wrote: post #321

Earlier quoted context omitted.

> Or typing 'o' + '¨' to produce 'ö' but realizing you wanted to type 'ô', there just one backspace press would revert it to 'o' again and you could press '^' to get the 'ô'. This is a good example because in German I would expect 'o' + '¨' + to leave no character at all while in French I would expect 'e' + '`' + to leave the e behind because in my mind it was a typo. The rendering of brahmic- and arabic-derived scri…

But typing "ö" (e.g. swiss keyboard) and pressing delete & getting an o would be annoying af

Definitely agree with that! I use a US kbd (incl on phone) no matter what language I’m writing in. A little annoying but switching kbd layouts is more disruptive for me.

Re: What every software developer must know about Unicode in 2023

#354

Post by lelanthran » Mon, Oct 02, 2023, 9:36 PM UTC

JohnFen wrote: post #340

lelanthran wrote: post #256

Earlier quoted context omitted.

> What do you mean by "work"? That you can store arbitrary bytes in a string? That's a pretty low bar. That's all that's needed for a backend language. The backend does not need to understand, or even acknowledge the existence, of grapheme clusters. Because the frontend is already having to understand all of this, it should be normalising any multi-codepoint ambiguous cluster anyway.

The backend never needs to do things like figure out how long a string is or search for one string in a database of other strings?

> The backend never needs to do things like figure out how long a string

Not as measured by clusters, no.

> search for one string in a database of other strings?

Hence I said "normalisation". The frontend already has to do all the unicode twiddling, it may as well normalise the input too.

Re: What every software developer must know about Unicode in 2023

#355

Post by ninkendo » Mon, Oct 02, 2023, 9:38 PM UTC

TacticalCoder wrote: post #292

> For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points! It's a poor and misleading example for it is definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there (french is the language where 'é' is the most common). 'é' is U+00F9, one codepoint, definitely not two. Now you could say: but it…

> Unicode the complete, total and utter clusterfuck that it is.

Yikes, does it really deserve that much derision? They’re trying to standardize all written human language here. I think they’ve done a fantastic job. Pre-Unicode you had to worry about what code page a document had, and computers from different countries couldn’t interoperate. The work the consortium does is hugely important, and every decision has extremely complex tradeoffs. Composed characters makes a lot of sense, and there’s a lot of case to be made that it was the right call. The attitude of “this one thing I don’t like makes the whole thing a complete clusterfuck” is something I wish fewer engineers would have.

Re: What every software developer must know about Unicode in 2023

#356

Post by astrange » Mon, Oct 02, 2023, 9:38 PM UTC

lelanthran wrote: post #256

uxp8u61q wrote: post #242

Earlier quoted context omitted.

What do you mean by "work"? That you can store arbitrary bytes in a string? That's a pretty low bar.

> What do you mean by "work"? That you can store arbitrary bytes in a string? That's a pretty low bar. That's all that's needed for a backend language. The backend does not need to understand, or even acknowledge the existence, of grapheme clusters. Because the frontend is already having to understand all of this, it should be normalising any multi-codepoint ambiguous cluster anyway.

It does if it ever wants to trim, summarize, sort or compare equal a string.

Re: What every software developer must know about Unicode in 2023

#357

Post by astrange » Mon, Oct 02, 2023, 9:39 PM UTC

pif wrote: post #193

uxp8u61q wrote: post #189

Earlier quoted context omitted.

This kind of assertiveness leads to garbage like C++ still not supporting UTF8 properly in 2023. My name contains diacritics. I am so, so, so tired of trying to work around information systems - not just web frontends - designed by people who don't care or worse, don't want to care. "Web" programmers can care all they want about Unicode, but if the backend people didn't deal properly with text encoding, then somethin…

> if the backend people didn't deal properly You are right. It's not a frontend/backend issue. It's a "for human" vs "not for human" issues. Personal names must be treated in an international-friendly manner. >> There are plenty of software realms where ASCII not only is enough, but it actually MUST be enough. > > Name one Joel himself described an example: > It would be convenient if you could put the Content-Type o…

HTTP does support content-type tags and Unicode in URLs. Which funny enough comes in two different encodings, punycode and percent escapes.

Re: What every software developer must know about Unicode in 2023

#358

Post by anymouse123456 » Mon, Oct 02, 2023, 9:43 PM UTC

dekken_ wrote: post #2

Am I supposed to hate this website, cause I kinda do

FWIW - Right Click, Inspect. There's a div with an attribute, "pointers" in the body root.

Deleting that makes the while thing a lot less stressful.

Re: What every software developer must know about Unicode in 2023

#359

Post by __d » Mon, Oct 02, 2023, 9:45 PM UTC

qwerty456127 wrote: post #23

> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to? Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11. Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last…

I write daily in US English, Australian English, and Austrian German. Most of the time, a specific document is in one dialect/language or another: not mixed, although sometimes that's not true.

I can understand that the conflation of spelling, word choices, time and date formatting, default paper sizes, measurement units, etc, etc, is convenient, and works a lot of the time, but it really doesn't work for me at all.

That said, I appreciate that I occupy a very small niche.

Re: What every software developer must know about Unicode in 2023

#360

Post by hot_gril » Mon, Oct 02, 2023, 9:47 PM UTC

bigstrat2003 wrote: post #104

badcppdev wrote: post #57

Earlier quoted context omitted.

To save other people the google: Tengwar is probably not in unicode because it is a fictional script from a book.

Honestly, I wouldn't have thought that would be an issue to the Unicode folks. They have already allowed things (emoji) that have no place being in the standard, as they aren't even text .

I feel like Apple pushed the consortium to add a ton of useless emojis for whatever their own reasons were.