Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

541–550 of 572 posts

Re: What every software developer must know about Unicode in 2023

#541

Post by b3morales »

astrange wrote: post #380
b3morales wrote: post #363

Earlier quoted context omitted.

> There's no reason to privilege any one of these, and Swift doesn't do this. Strange thing to say: Swift String count property is the count of extended grapheme clusters. The documentation is explicit: > A string is a collection of extended grapheme clusters , which approximate human-readable characters. [emphasis in original]

The length/count property was added after people asked for it, but it wasn't originally in the String revamp, and it provides iterators for all of the above. .count also only claims to be O(n) to discourage using it.

That was almost seven years ago now. It has been the String API twice as long as it has not been the API.

Re: What every software developer must know about Unicode in 2023

#542

Post by rishav_sharan »

TacticalCoder wrote: post #292

> For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points! It's a poor and misleading example for it is definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there (french is the language where 'é' is the most common). 'é' is U+00F9, one codepoint, definitely not two. Now you could say: but it…

Unicode being an utter cluster fuck is my takeaway from this as well. Are there any alternatives to Unicode? Maybe something which has a single grapheme code point and uses utf32 for the encoding. Or anything else saner than what we have today?

Re: What every software developer must know about Unicode in 2023

#543

Post by rstuart4133 »

lifthrasiir wrote: post #456
rstuart4133 wrote: post #445

Earlier quoted context omitted.

Excellent reply. > You can easily have more than 10M code points in this way. The current set of Hangul syllables, precomposed or not, is 125 * 95 * 138 = 1,638,750 characters. Latin characters with at most 3 combining marks (known to exist in the wild) would be probably in the same order of magnitude. Maybe now you can try, thanks to the computing power and all the information, but in 1990? Fat chance. It can be mad…

I appreciate your reply, which I never expected in this situation. If my understanding is correct, your thesis is that Unicode should be hidden from application programmers as much as possible, much like the fact that GC hides memory management so to say. Not to say Unicode is bad or even shouldn't exist at all, but something like that it has to be abstracted away. This is a much more reasonable than most (quote-unqu…

Thanks again for the very informative reply.

> You may still argue that Unicode algorithms are designed for the human text encoded in strings. That's a very nuanced argument, because one can also argue that they are the best effort approximation of human text operations for strings, in which case they are not the human text operations themselves. For example many languages have a case conversion operation over strings,

That is ... very nuanced indeed. Case folding does look to be difficult issue in it's own right, but it's not something I do a lot and besides even 20 years ago it was deferred to a library function (str.lower() or whatever the language provides).

The issue the original article correctly says every app trips over is neither nuanced, nor uncommon. It's boring stuff like handling backspace or left arrow cursor movement - stuff programmer have to do all the time, and that is difficult to put in a library. As a consequence they get it wrong over and over again. Unicode representing a grapheme with a single code point would fix that.

As you say I'm not sure there is a simple change you could make to Unicode that renders case folding or the other problems you describe easy. To me that's a strong hint it's not the right place to address those problems.

Re: What every software developer must know about Unicode in 2023

#544

Post by loeg »

fomine3 wrote: post #414
loeg wrote: post #333

Pretty clearly, "every software developer" doesn't need to understand Unicode with this level of familiarity, much like "every programmer" doesn't need to know the full contents of the 114 page Drepper paper. For example, I work on a GUID-addressed object store. Everything is in term of bytes and 128-bit UUIDs. Unicode is irrelevant to everyone on my team, and most adjacent teams. There is lots of software like this.

Every programmer don't have to remember all things in this article, but they should remember that Unicode (or text system in the wild) is actually complex so they should research as needed.

Sure, that's a reasonable take.

Re: What every software developer must know about Unicode in 2023

#545

Post by ssokolow »

dystroy wrote: post #498
ssokolow wrote: post #438

> Before comparing strings or searching for a substring, normalize! ...and learn about the TR39 Skeleton Algorithm for Unicode Confusables. Far too few people writing spam-handling code know about that thing. (Basically, it generates matching keys from arbitrary strings so that visually similar characters compare identical, so those Disqus/Facebook/etc. spam messages promoting things like BITCO1N pump-and-dumps or us…

But not all normalizations are done to fight spam, not all of them should be interested in visual similarity. I normalize strings in searches not because of bad intents but because for all user related purposes "Comunicações" and "Comunicações" are the same, their different encodings being more of an accident.

*nod* ...and stemming is that taken to a greater extreme.

I was just pointing out that Unicode itself has various forms of normalization and normalization-adjacent functionality that people are far too unaware of.

Re: What every software developer must know about Unicode in 2023

#546

Post by toastercat »

tqwhite wrote: post #525
toastercat wrote: post #418

Earlier quoted context omitted.

1. Not every browser has reader mode 2. I don't think it's a very good joke to post long-form content on your blog with the expectation that it's basically unreadable without a reader mode. > The fact that people seem to take it seriously says something about the contemporary state of webdesign :) Mind expanding and what it says exactly about contemporary web design? Whether I take it seriously or not doesn't change…

If you don't have reader mode, get a new browser. Don't tell him to make it boring for all the rest of us who behave normally.

Boring and readable are not the same thing. Also, you can edit your comments on HackerNews

Re: What every software developer must know about Unicode in 2023

#547

Post by anthk »

AlphaCerium wrote: post #453
anthk wrote: post #373

Earlier quoted context omitted.

Use the reader mode. Or if you are under GNU/Linux, use Links/Lynx.

Not everyone runs Linux, and not every browser has a reader mode. This should not be the solution. There should definitely be an option to disable all these features, especially the dark mode toggle, that one's a fun premise, but horrific for usability.

True; but Links/Lynx exists for Windows, too. Or Netsurf. At least there are alternatives to choose. But you are right, the web sucks.

Re: What every software developer must know about Unicode in 2023

#549

Post by kalleboo »

paledot wrote: post #499
Symbiote wrote: post #463

Earlier quoted context omitted.

Note that in Swedish they are considered letters, and in Danish and Norwegian Æ, Ø and Å are letters.

Letters which are sorted separately from what we'd think of as the base characters in English (they appear at the end of the alphabet, as W X Y Z Æ Ø Å, with C often omitted in Norwegian). By contrast, my French dictionary has énorme nestled between enorgueillir and enquiérir. (Looking for an example does underscore some of the patterns in the language: page after page of ét~ with only a few et~ and one êt~ among the…

Similarly in Swedish, W was not considered a letter but just a variant of V, so in phone books etc all the W names were mixed in with V names. This was changed in 2006 due to an increase in English loanwords.

Re: What every software developer must know about Unicode in 2023

#550

Post by mike_hearn »

davidmurdoch wrote: post #533
mike_hearn wrote: post #519

Earlier quoted context omitted.

Presizing buffers, initializing for loop counts ...

Unrelated to this post, but you suggested ( https://news.ycombinator.com/item?id=37381390 ) I use your company hydraulic.dev for my electron build. I ultimately gave up on it, then someone from node-gyp comment on an issue I opened about it, and provided the solution: https://github.com/electron-userland/electron-builder/issues... Just wanted to let you know in case it's a gotcha you might not be aware of that might…

Thanks for the tip! Sorry to hear you gave up on it, it'd be really appreciated if you could email me with some info as to what problems you hit. We're improving our Electron support at the moment (adding ASAR support and so on), so if there's low hanging fruit it'd be good to know where to look.

The bug you linked to is a bit confusing, it seems to be a bug in electron-builder (or node-gyp), not Conveyor?

Post reply on HN