Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

221–230 of 572 posts

Re: What every software developer must know about Unicode in 2023

#221

Post by [deleted] »

orphea wrote: post #195
pif wrote: post #152

Earlier quoted context omitted.

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".

[deleted]

Re: What every software developer must know about Unicode in 2023

#223

Post by fiedzia »

thyselius wrote: post #202

Wonderful to learn more about Unicode. Does anyone know how to write a function (preferably in swift) to remove emoji? This is surprisingly hard (if the string can be any language, like English or Chinese). There’s been multiple attempts on Stackoverflow but they’re all missing some of them, as Unicode is so complex.

I haven't tried but use libicu (icu). Split text into graphemes and remove anything starting with codepoints that has Zsey script. There should be swift bindings.

Re: What every software developer must know about Unicode in 2023

#224

Post by raphlinus »

jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

Agreed. And one more consideration is that (extended) grapheme cluster boundaries vary from one version of Unicode to another, and also allow for "tailoring." For example, should "อำ" be one grapheme cluster or two? It's two on Android but one by Unicode recommendation and is the behavior on mac. So in applications where a query such as length needs to have one definitive answer which cannot change by context, counting (extended) grapheme clusters is the wrong way to go.

Re: What every software developer must know about Unicode in 2023

#225

Post by hot_gril »

torstenvl wrote: post #219
hot_gril wrote: post #180

Earlier quoted context omitted.

iMessage has had several vulnerabilities related to this. Whatever the difficulties are, even Apple can't handle them sometimes.

I'm very skeptical, but willing to be proven wrong. What's the CVE?

First that comes to mind is the "effective power" one, https://nvd.nist.gov/vuln/detail/cve-2015-1157 There's also the "black dot" one, can't find the CVE though.

Re: What every software developer must know about Unicode in 2023

#226

Post by aembleton »

coldpie wrote: post #168
toastercat wrote: post #119

Earlier quoted context omitted.

Anytime tonsky's site gets posted here, I'm reminded by how awful it is, which is ironic given his UI/UX background. The site's lightmode is a blinding saturated yellow, and if you switch into darkmode, it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.

Works like a normal website with JavaScript disabled. I didn't even know it did fancy junk until reading the comments here. NoScript saves the day again! I don't know how people can browse the web without it.

By using reader mode

Re: What every software developer must know about Unicode in 2023

#227

Post by fragmede »

zzzeek wrote: post #30
davidham wrote: post #10

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

yeah....why on earth would someone want their webpage to do this, especially if they have text they'd presumably want you to read?

Have you ever read with other people, like in school or a book club, or been somewhere that there were other people around? It's an interesting move by the author; the loneliness epidemic hasn't gone unnoticed.

eg https://www.npr.org/2023/05/02/1173418268/loneliness-connect...

Re: What every software developer must know about Unicode in 2023

#228

Post by [deleted] »

masklinn wrote: post #123
qwerty456127 wrote: post #23

> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to? Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11. Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last…

> English (USA), English (UK) > This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. While that is generally (though not always) true, I would assume it's really a stand in for the much more relevant zh locales. It is also rather relevant to es locales (america spanish has diverged quite a b…

[deleted]

Re: What every software developer must know about Unicode in 2023

#229

Post by notatoad »

qwerty456127 wrote: post #23

> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to? Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11. Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last…

> I doubt many English speakers care to distinguish between English dialects

I think you'd be surprised how many english (UK) people will get pissed off when their spell-checker starts removing the "u" from colour or flavour, or how many English (US) people get pissed off when the spellchecker starts suggesting random "u"s to words.

additionally to that, locale isn't just about language. English (US) and English (UK) decides whether your dates get formatted DD-MM-YY or MM-DD-YY, whether your numbers have the thousands broken by commas or spaces, and a host of other localization considerations with a lot more significance than just the dialect of english.

Re: What every software developer must know about Unicode in 2023

#230

Post by rkagerer »

"the definition of graphemes changes from version to version"

In what twisted reality did someone think this a good idea?

Doesn't it go against the whole premise of everyone in the world agreeing on how to represent a meaningful unit of text?

"What’s sad for us is that the rules defining grapheme clusters change every year as well. What is considered a sequence of two or three separate code points today might become a grapheme cluster tomorrow! There’s no way to know! Or prepare!"

"Even worse, different versions of your own app might be running on different Unicode standards and report different string lengths!"

Post reply on HN