Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

241–250 of 572 posts

Re: What every software developer must know about Unicode in 2023

#241

Post by uxp8u61q »

pif wrote: post #193
uxp8u61q wrote: post #189

Earlier quoted context omitted.

This kind of assertiveness leads to garbage like C++ still not supporting UTF8 properly in 2023. My name contains diacritics. I am so, so, so tired of trying to work around information systems - not just web frontends - designed by people who don't care or worse, don't want to care. "Web" programmers can care all they want about Unicode, but if the backend people didn't deal properly with text encoding, then somethin…

> if the backend people didn't deal properly You are right. It's not a frontend/backend issue. It's a "for human" vs "not for human" issues. Personal names must be treated in an international-friendly manner. >> There are plenty of software realms where ASCII not only is enough, but it actually MUST be enough. > > Name one Joel himself described an example: > It would be convenient if you could put the Content-Type o…

> The content of a webpage is required to be expressed in every supported language, but the HTTP protocol must not. And it would make no sense at all to add internationalization to intra-machines protocol, where ASCII is enough and has been enough for decades.

I guess no URLs with funny characters then. "GET /profile/renée" => 500 error, woohoo.

> And if someone complains that ASCII only supports English, well... suck it up! I'm Italian and work in French, still I hate when a colleague sneaks in a comment not in English. Professional software development happens in English.

Get over yourself, a lot of professional development happens in languages other than English.

Re: What every software developer must know about Unicode in 2023

#242

Post by uxp8u61q »

lelanthran wrote: post #216
uxp8u61q wrote: post #189

Earlier quoted context omitted.

This kind of assertiveness leads to garbage like C++ still not supporting UTF8 properly in 2023. My name contains diacritics. I am so, so, so tired of trying to work around information systems - not just web frontends - designed by people who don't care or worse, don't want to care. "Web" programmers can care all they want about Unicode, but if the backend people didn't deal properly with text encoding, then somethin…

> This kind of assertiveness leads to garbage like C++ still not supporting UTF8 properly in 2023. My name contains diacritics. UTF8 encoded diacritics work just fine in C++.

What do you mean by "work"? That you can store arbitrary bytes in a string? That's a pretty low bar.

Re: What every software developer must know about Unicode in 2023

#243

Post by aksss »

notatoad wrote: post #229
qwerty456127 wrote: post #23

> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to? Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11. Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last…

> I doubt many English speakers care to distinguish between English dialects I think you'd be surprised how many english (UK) people will get pissed off when their spell-checker starts removing the "u" from colour or flavour, or how many English (US) people get pissed off when the spellchecker starts suggesting random "u"s to words. additionally to that, locale isn't just about language. English (US) and English (UK)…

I worked for BP for a while (well, as a contracted coder) and I got quite used to the UK spell check correcting everything to its idiom. Everything seemed wrong once I returned a world that dismissed the value of the letter 'U' and preferred the letter 'Z' over 'S'. Also missed the normalizing of drinking beer at lunch.

Re: What every software developer must know about Unicode in 2023

#244

Post by P-Nuts »

swores wrote: post #136
gwervc wrote: post #118

Earlier quoted context omitted.

The circled digits as code points are very nice to have precisely because they are available in applications that don't support them otherwise... which is actually most of the software I can think of (Notepad, Apple Notes, chat applications, most websites, etc).

Can you write them with iOS keyboard? Or when you say Apple Notes and chat apps you just mean from desktop? Edit ①: seems the answer is not with the default iOS keyboard, but possible to paste it and perhaps possible with a third party keyboard that I'm not keen on trying (unless I hear of a keyboard that's both genuinely useful / better than default, and that doesn't send keystrokes to the developer - though I can't…

You can type ① with the UniChar keyboard app on iOS. It at least claims it doesn’t transmit information. As it’s only useful for special characters I don’t worry because I can’t use it for normal typing anyway.

https://unichar.app

Re: What every software developer must know about Unicode in 2023

#245

Post by arp242 »

d3w4s9 wrote: post #214
arp242 wrote: post #77

Earlier quoted context omitted.

> Unicode doubled down with composed Emojis. Not just emojis, in general I believe Unicode has just said they're not going to add new pre-composed characters and that using combining characters is the Right Way™ to do things (well, the only way for newer scripts). One of the downsides of writing down specifications is that they tend to attract people with Very Strong Opinions on the One And Only Right Way and will ar…

The first time I heard that Unicode would support emoji, I knew it would be a recipe for disaster. And I definitely was not disappointed.

I mean, I don't dislike the concept personally. I actually really hate how HN strips them.

But the technical implementation? Yeah, that could have gone a lot better IMHO.

One must also wonder if some things really had to be added in the first place, e.g. for people kissing it's:

  (person|man|woman)(skin-tone)? ZWJ  ZWJ  ZWJ (person|man|woman)(skin-tone)?
This is NOT a complaint about that they added diversity as such, in principle I'm all for that, it's just that few seem to actually use these emojis, and both in terms of code and UI it all gets pretty complex; there's 98 combinations to choose from here.

I don't really get why or or isn't enough. That's actually what most people seem to use anyway, because who finds it convenient to pick all the correct genders and skin tones from the UI for both people?

Re: What every software developer must know about Unicode in 2023

#246

Post by nottorp »

nwellnhof wrote: post #47
tialaramex wrote: post #42

Earlier quoted context omitted.

The writing systems were already like this when we got them. Unicode's "total mess" mostly just reflects that. Of course it would be convenient for you, the programmer, if the users wanted the software to do whatever was easiest for you, but obviously they want what's easiest for them, not you.

Name one writing system where you really need character composition. Even if there is one, these special cases should be handled outside of Unicode.

The problem is not that you need character composition for some writing systems. It's that there are no rules that would help with everything having an unique representation internally.

Even "put the code points forming the composed character in descending numerical order" would be better than nothing. If it was there from the start.

However, the Unicode commitee is too busy adding new emojis to make their standard sane.

Re: What every software developer must know about Unicode in 2023

#247

Post by zzo38computer »

kajaktum wrote: post #176

I am torn between supporting all languages (which easily leaks into supporting emojis) versus just using the 90~ Latin characters as the lingua franca. Look, I would love to be able to read/write Sanskrit, Arabic, Chinese, Japanese etc and share those content and have everyone render and see the same thing. The problem is that I feel like most of these are: 1. a kind of an open problem 2. very subjective 3. very, ver…

> I am torn between supporting all languages (which easily leaks into supporting emojis) versus just using the 90~ Latin characters as the lingua franca.

I don't want to support emoji either (and, I don't want emoji on my computer), although in some cases, if it is really necessary to be supported, they could be implemented just as text characters instead of as colourful emoji, anyways.

For many purposes (e.g. computer codes) ASCII is good enough (and actually even can be better since it avoids the security problems of using Unicode). (Sometimes, character sets other than ASCII can be used, e.g. APL character set for APL programming.)

> Look, I would love to be able to read/write Sanskrit, Arabic, Chinese, Japanese etc

I also would, but Unicode is bad enough that I would use other ways of doing such a thing when possible (even writing my own programs, etc). (If a program insists on Unicode, I might just use ASCII only anyways, or write my own program)

Not everyone necessarily need to see the same thing (if it is a text, rather than pictures of the text), although, the suitable character sets for that language which can be in use (and with fix pitch if necessary, etc), to auto select a suitable fonts for your computer by the reader's preference.

So, I prefer to support all languages (where applicable; sometimes it isn't), without using Unicode.

Re: What every software developer must know about Unicode in 2023

#248

Post by nottorp »

arp242 wrote: post #245
d3w4s9 wrote: post #214

Earlier quoted context omitted.

The first time I heard that Unicode would support emoji, I knew it would be a recipe for disaster. And I definitely was not disappointed.

I mean, I don't dislike the concept personally. I actually really hate how HN strips them. But the technical implementation? Yeah, that could have gone a lot better IMHO. One must also wonder if some things really had to be added in the first place, e.g. for people kissing it's: (person|man|woman)(skin-tone)? ZWJ ZWJ ZWJ (person|man|woman)(skin-tone)? This is NOT a complaint about that they added diversity as such, i…

> I actually really hate how HN strips them.

Oh. So that's why HN discussion always looks sane. They strip the pollution.

Re: What every software developer must know about Unicode in 2023

#249

Post by torstenvl »

hot_gril wrote: post #225
torstenvl wrote: post #219

Earlier quoted context omitted.

I'm very skeptical, but willing to be proven wrong. What's the CVE?

First that comes to mind is the "effective power" one, https://nvd.nist.gov/vuln/detail/cve-2015-1157 There's also the "black dot" one, can't find the CVE though.

That seems like a truncation + display issue though, not a normalization issue.

https://www.reddit.com/r/apple/comments/37e8c1/malicious_tex...

In fact, I don't know that there's any reason to believe normalization happens at all in the process of executing this.

Post reply on HN