Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

191–200 of 572 posts

Re: What every software developer must know about Unicode in 2023

#191

Post by superq »

ericmcer wrote: post #135
toastercat wrote: post #119

Earlier quoted context omitted.

Anytime tonsky's site gets posted here, I'm reminded by how awful it is, which is ironic given his UI/UX background. The site's lightmode is a blinding saturated yellow, and if you switch into darkmode, it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.

I don't think he added moving cursors all over the page because he thought it was good UI/UX, he knows what he is doing.

This is seemingly self-contradictory. Perhaps you could explain your reasoning further?

Re: What every software developer must know about Unicode in 2023

#192

Post by w10-1 »

A real question is why IBM, Apple, and Microsoft poured millions into developing the unicode standard instead of treating character encoding like file formats as a venue for competition.

IBM and Apple in the early 1990's combined in Taligent to try to beat MS NT, but failed. But a lot of internationalization came out of that and was made open, at the perfect time for Java to adopt it.

Interestingly it wasn't just CJK but Thai language variants that drove much of the flexibility in early unicode, largely because some early developers took a fancy to it.

When you look at the actual variety in written languages, Unicode grapheme/code-point/byte seems rather elegant.

We're in the early days of term vectors, small floats, and differentiable numerics (not to mention big integers). Are lessons from the history of unicode relevant?

Re: What every software developer must know about Unicode in 2023

#193

Post by pif »

uxp8u61q wrote: post #189
pif wrote: post #162

> The minimum every software developer must know about Unicode Just a nitpick... Once more, as it is typical on HN, web programming is confused with the entire universe of software development. There are plenty of software realms where ASCII not only is enough, but it actually MUST be enough.

This kind of assertiveness leads to garbage like C++ still not supporting UTF8 properly in 2023. My name contains diacritics. I am so, so, so tired of trying to work around information systems - not just web frontends - designed by people who don't care or worse, don't want to care. "Web" programmers can care all they want about Unicode, but if the backend people didn't deal properly with text encoding, then somethin…

> if the backend people didn't deal properly

You are right. It's not a frontend/backend issue. It's a "for human" vs "not for human" issues. Personal names must be treated in an international-friendly manner.

>> There are plenty of software realms where ASCII not only is enough, but it actually MUST be enough. > > Name one

Joel himself described an example:

> It would be convenient if you could put the Content-Type of the HTML file right in the HTML file itself, using some kind of special tag. Of course this drove purists crazy… how can you read the HTML file until you know what encoding it’s in?! Luckily, almost every encoding in common use does the same thing with characters between 32 and 127, so you can always get this far on the HTML page without starting to use funny letters:

The content of a webpage is required to be expressed in every supported language, but the HTTP protocol must not. And it would make no sense at all to add internationalization to intra-machines protocol, where ASCII is enough and has been enough for decades.

And if someone complains that ASCII only supports English, well... suck it up! I'm Italian and work in French, still I hate when a colleague sneaks in a comment not in English. Professional software development happens in English.

Re: What every software developer must know about Unicode in 2023

#194

Post by Findecanor »

jordanrobinson wrote: post #51

Anyone know what the story is behind the "Weird Emoji" around 140000 on the map?

The E0000-E007F block is the "Tags" block, which is used for flag emojis.

But there is not a code for each flag. Instead there is a code for each ASCII character. A flag sequence is formed from U+1F3F4 (Black Flag), followed by at least two tags that form a country/region code, and then U+E007F (End tag).

So, yes this is weird, because the emoji is dependent on the decoder. It was made this way to keep Unicode independent of geopolitics.

Read more: https://en.wikipedia.org/wiki/Tags_(Unicode_block)>

Re: What every software developer must know about Unicode in 2023

#195

Post by orphea »

pif wrote: post #152
jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".

Re: What every software developer must know about Unicode in 2023

#197

Post by tiltowait »

d11z wrote: post #160
swores wrote: post #136

Earlier quoted context omitted.

Can you write them with iOS keyboard? Or when you say Apple Notes and chat apps you just mean from desktop? Edit ①: seems the answer is not with the default iOS keyboard, but possible to paste it and perhaps possible with a third party keyboard that I'm not keen on trying (unless I hear of a keyboard that's both genuinely useful / better than default, and that doesn't send keystrokes to the developer - though I can't…

Speaking of third party keyboards, I’m still upset about what happened to Nintype[0]. I’ve never ever been able to type faster on mobile than with it’s intuitive hybrid input style of sliding and tapping, paired with AI that was actually good. It used to be quite performant, fully customizable, and it worked beautifully as a replacement for default on jailbroken iOS. Today, it’s buggy $5 abandonware that only makes m…

Nintype was absolutely incredible. I still open it every now and then after an iOS update in the vain hope some system change made it less buggy.

Re: What every software developer must know about Unicode in 2023

#198

Post by kdmccormick »

orphea wrote: post #195
pif wrote: post #152

Earlier quoted context omitted.

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".

But then if I type "á" directly (through, say, a mobile keyboard) and hit backspace, I'd get "a", which doesn't seem terrible but does feel a little off.

Seems like the right answer for codepoints vs graphemes, unfortunately, is dependent on the context.

Re: What every software developer must know about Unicode in 2023

#200

Post by bombela »

orphea wrote: post #195
pif wrote: post #152

Earlier quoted context omitted.

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".

I expect to delete the character "á". And I prefer consistency too so I expect "œ" and "" and "" to be deleted as one unit.

edit: emojis are filtered by HN

Post reply on HN