Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

561–570 of 572 posts

Re: What every software developer must know about Unicode in 2023

#561

Post by [deleted] »

jfultz wrote: post #172
pif wrote: post #152

Earlier quoted context omitted.

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

A couple of cases I'm aware of... * Coding ligatures often display as a single glyph (maybe occupying a single-width character space, or maybe spread out over multiple spaces), but are composed of multiple glyphs. The ligature may "look" like a single character for purposes of selection and cursoring, but it can act like multiple characters when subject to backspacing. * Similarly, I've seen keyboard interfaces for v…

[deleted]

Re: What every software developer must know about Unicode in 2023

#563

Post by scruss »

davidham wrote: post #10

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

as someone with a visual processing disorder, this is like having a page scream at me. Repeatedly. Never do this

Re: What every software developer must know about Unicode in 2023

#564

Post by WillAdams »

neerajsi wrote: post #442
uxp8u61q wrote: post #182

Earlier quoted context omitted.

That's unrelated to unicode. The checkmark symbol just isn't in the Arial font, so Word just falls back to a font that has it - Segoe UI. You've found a bug where Word still thinks it's Arial. But this is something that would happened no matter what encoding you choose for your characters.

I don't know this for a fact, but it's possible that the text run is logically considered to be Arial and the fallback could be handled as just a rendering step, rather than being encoded in the document. Doing it that way could allow the text to render on different versions of Arial, some of which do have a checkbox char, at the risk of the appearance and layout changing depending on which fonts are installed.

The weird thing is, it didn't work thus for other numbers for used for this or similar characters.

Re: What every software developer must know about Unicode in 2023

#565

Post by elcaro »

Another Unicode article that mentions Swift, but not Raku :(

Raku's Str type has a `.chars` method that counts graphemes. It has a separate `.codes` method to count codepoints. It also can do O(1) string indexing at the grapheme level.

That Zalgo "word" example is counted as 4 chars, and the different comparisons of "Å" are all True in Raku.

You can argue about the merits of it's approach (indeed several commenters here disagree that graphemes are the "one true way" to count characters), but it feels lacking to not at least _mention_ Raku when talking about how different programming languages handle Unicode.

Re: What every software developer must know about Unicode in 2023

#566

Post by aksss »

denton-scratch wrote: post #487
aksss wrote: post #243

Earlier quoted context omitted.

I worked for BP for a while (well, as a contracted coder) and I got quite used to the UK spell check correcting everything to its idiom. Everything seemed wrong once I returned a world that dismissed the value of the letter 'U' and preferred the letter 'Z' over 'S'. Also missed the normalizing of drinking beer at lunch.

> Also missed the normalizing of drinking beer at lunch. Perhaps you're an old-timer? I worked in the city in the early 80s; lunch in the pub was routine, and sometimes required. By the end of the 80s, that was at best frowned on. Over the last 20 years, having alcohol on your breath after lunch would have been a disciplinary issue, unless you were entertaining a client, at least in the places I worked.

It's likely because I was in the exploration frontier of Alaska. This was late nineties, probably an operation run by people who worked in the city in the '80's and who could continue the old ways far away from the social glare of the head office. :D

Re: What every software developer must know about Unicode in 2023

#567

Post by johnnyanmac »

toastercat wrote: post #119
davidham wrote: post #10

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

Anytime tonsky's site gets posted here, I'm reminded by how awful it is, which is ironic given his UI/UX background. The site's lightmode is a blinding saturated yellow, and if you switch into darkmode, it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.

>it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.

not even a proper flashlight. it updates when the mouse moves, so you're SOL if you scroll on desktop.

Re: What every software developer must know about Unicode in 2023

#568

Post by TRiG_Ireland »

lucideer wrote: post #470
phantom784 wrote: post #313

Earlier quoted context omitted.

I guess it's a question as to how many varieties of spelling you want to make available as "translations" in software (e.g. color vs colour, tire vs tyre). There's plenty of regional variants just within the US, but "en_us" covers the whole country.

That's a fair point - even in tiny tiny Ireland there's many regional dialects, with larger countries there'll typically be far more. I guess the simple answer to that is: how much interest is there in maintenance. I don't think there's any compelling reason not to create something: if there's insufficient interest in maintenance that's an imperfect but reasonable proxy for utility. I'm not aware of any maintained en…

For both Norwegian and Chinese there are two separate written standards, along with the far more varied spoken dialects/languages.

Re: What every software developer must know about Unicode in 2023

#569

Post by kalinf »

> Among them is assigning the same code point to glyphs that are supposed to look differently, like Cyrillic Lowercase K and Bulgarian Lowercase K (both are U+043A).

This is nonsense, Bulgaria has been using the Cyrillic alphabet sinse its creation in … Bulgaria!

What you’ve shown is two different fonts, and both renderings are perfectly fine in Bulgaria.

Read up more about it on wikipedia: https://en.wikipedia.org/wiki/Bulgarian_alphabet

Re: What every software developer must know about Unicode in 2023

#570

Post by ycomfakeuser »

There is another 'modern' language that does utf8 right and has done it right for a long time. I know it's mostly fallen out of favour, but we're still out here: Perl.

$ perl -wle 'use utf8; print length("");' 1

Without use utf8: $ perl -wle 'print length("");' 3

It's funny: after Perl fell out of favour, is when it got all its best stuff. It's still my preferred language for just about everything.

Post reply on HN