What every software developer must know about Unicode in 2023 • Hacker News Board

Re: What every software developer must know about Unicode in 2023

#481

Post by mananaysiempre » Tue, Oct 03, 2023, 1:08 PM UTC

hunter2_ wrote: post #390

riggsdk wrote: post #374

Earlier quoted context omitted.

I realize that the editor would be the system to keep track of how the character was entered for this to work. If you made the character from a single keypress it would only make sense that backspace also undid the entire character. Only if you created the character from multiple keypresses it would make sense to "undo" only part of it with backspace (at least until you move away from the character).

> make sense to "undo" only part of it with backspace I'm not sure that ever really makes sense: it would be a misnomer if "backspace" didn't bring you "back" some amount of horizontal "space," I reckon. This logic holds up not only for cases like ö and emoji (where I'd hope the whole grapheme disappears), but also for scenarios like if one types and an ligature appears, where I'd hope only the disappears: that's fin…

“Backspace” is already a misnomer: the original intention was for it to move the caret one position back, the other way around from the usual space, thus enabling overstriking (whence also the characters _ ^ ` , unheard of before the typewriter age). You can still see this used by the troff|less internals of man, which encode underline and bold as _a and aa respectively.

Re: What every software developer must know about Unicode in 2023

#482

Post by coldpie » Tue, Oct 03, 2023, 1:09 PM UTC

redder23 wrote: post #413

coldpie wrote: post #168

Earlier quoted context omitted.

Works like a normal website with JavaScript disabled. I didn't even know it did fancy junk until reading the comments here. NoScript saves the day again! I don't know how people can browse the web without it.

I never understood how people can browser the we WITH IT. Even 10 years ago. today more then ever basically every website needs JS to work properly. I basically never come across a page where I have the urge to disable JS. I have a large list of adblock lists active that also help getting rid of cookie banners and other shit. I can not imagine manually approving JS for every site. And again doing the inverse and have…

Many sites don't need JS at all, like the OP of this thread for example. For a lot of sites, disabling JS actually gives a better experience than leaving it enabled, again like the OP of this thread. It's a trade-off, but I find most uses of JS are so bad it's worth putting up with whitelisting. For example, I don't see cookie pop-ups, I don't see videos, disabling JS kills most of those stupid sticky headers that web designers love so much, and whatever too-clever crap the OP of this thread was doing is completely bypassed. The web is so, so much better with JS off by default.

For those sites that do need JS, NoScript's whitelist feature makes it quick & easy to fix. The first time I visit a new site, if it is obviously broken, then I whitelist the main domain. If that doesn't fix it, then I whitelist a couple likely-looking domains (often sites import JS from similar domains, or from common library domains). That's enough to get probably 90+% of websites working, while still leaving most garbage JS disabled. The remaining ~10% of websites that need a dozen domains whitelisted are probably not worth visiting anyway, so I just move on at that point. Or NoScript even lets you temp-whitelist everything for a given tab and just put up with the misery to get whatever I need from that one site. Since the whitelist persists forever, and I don't visit hundreds of different websites every day, after some time it becomes pretty rare that I need to whitelist more than one or two things per day.

You maintain an adblock blacklist, I maintain a NoScript whitelist. Not so different :)

Re: What every software developer must know about Unicode in 2023

#483

Post by denton-scratch » Tue, Oct 03, 2023, 1:10 PM UTC

l72 wrote: post #140

lucideer wrote: post #129

Earlier quoted context omitted.

As an Irish person, while we have en_IE which is great (and solves most of the problems you list re: Euro-centric defaults + English), I'd still quite like to have an even more broad / trans-language / "cosmopolitan" locale to use. I mainly type in English but occasionally other languages - I use a combination of Mac & Linux - macOS has an (off-by-default but enable-able) lang-changer icon in the tray that is handy e…

I write in multiple languages daily on Linux, including English, Russian, and Chinese. Switching keyboards (at least with gnome) is a simple super-space. While in my default (English) layout, it is easy enough to add in accents other characters using the compose key (right alt). So right-alt+'+a = á or right-alt+"+u = ü. I much prefer this over the long press as I can do it quickly and seamlessly without having to wa…

> So right-alt+'+a = á or right-alt+"+u = ü.

Not for me!

Right-alt+'+a = â

Right-alt+'+u = û

Re: What every software developer must know about Unicode in 2023

#484

Post by coldpie » Tue, Oct 03, 2023, 1:14 PM UTC

gpvos wrote: post #325

coldpie wrote: post #168

Earlier quoted context omitted.

Works like a normal website with JavaScript disabled. I didn't even know it did fancy junk until reading the comments here. NoScript saves the day again! I don't know how people can browse the web without it.

It is some time ago since I last used it, but I found that too many websites that I want to read require Javascript to even show you the main body of text, or a reasonable layout. Is that different now?

No. Just whitelist the main domain for sites that are obviously broken. Then try one or two likely subdomains if that's not enough. In the rare cases where it still wants more crap enabled, then it's usually not worth the effort, close tab and move on to something else. As you build up a whitelist over time, it becomes pretty rare that you need to interact with it more than a couple times per day. Yeah, it takes some effort, but it's worth it to nuke cookie banners and sticky headers and videos and all the other crap people do with JS.

Re: What every software developer must know about Unicode in 2023

#485

Post by kalleboo » Tue, Oct 03, 2023, 1:16 PM UTC

haberman wrote: post #364

jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

In that case, it sounds like `length` on Unicode strings simply shouldn't exist, since there is no obvious right answer for it. Instead there should be `codepointCount`, `graphemeCount`, etc.

There are basically 2 places where programmers mostly want the "length" of a string:

1. To save storage space or avoid pathological input, they want to limit the "length" of text input fields. E.g., not allow a name to be 4 KB long

2. To fit something on screen

Developers mostly used to western languages can approximate both with "number of letters", but the correct answers are

For 1. Limit to bytes to avoid people building infinite zalgo characters, but be intelligent about it - don't just crop the byte array not taking into account graphemes.

For 2. This sucks, especially for the web, but the only really correct answer here is to render it and check.

Did I miss any other common cases?

Re: What every software developer must know about Unicode in 2023

#486

Post by extraduder_ire » Tue, Oct 03, 2023, 1:17 PM UTC

posix86 wrote: post #329

gumby wrote: post #321

Earlier quoted context omitted.

> Or typing 'o' + '¨' to produce 'ö' but realizing you wanted to type 'ô', there just one backspace press would revert it to 'o' again and you could press '^' to get the 'ô'. This is a good example because in German I would expect 'o' + '¨' + to leave no character at all while in French I would expect 'e' + '`' + to leave the e behind because in my mind it was a typo. The rendering of brahmic- and arabic-derived scri…

But typing "ö" (e.g. swiss keyboard) and pressing delete & getting an o would be annoying af

It would probably expose the implementation too much, but if you wanted combined characters to be split apart, I would expect "ö" to be removed in one go, whereas an "o" with a combined diaeresis (No idea if it's called that, I got the name from the code chart) would take two backspaces.

Re: What every software developer must know about Unicode in 2023

#487

Post by denton-scratch » Tue, Oct 03, 2023, 1:20 PM UTC

aksss wrote: post #243

notatoad wrote: post #229

Earlier quoted context omitted.

> I doubt many English speakers care to distinguish between English dialects I think you'd be surprised how many english (UK) people will get pissed off when their spell-checker starts removing the "u" from colour or flavour, or how many English (US) people get pissed off when the spellchecker starts suggesting random "u"s to words. additionally to that, locale isn't just about language. English (US) and English (UK)…

I worked for BP for a while (well, as a contracted coder) and I got quite used to the UK spell check correcting everything to its idiom. Everything seemed wrong once I returned a world that dismissed the value of the letter 'U' and preferred the letter 'Z' over 'S'. Also missed the normalizing of drinking beer at lunch.

> Also missed the normalizing of drinking beer at lunch.

Perhaps you're an old-timer? I worked in the city in the early 80s; lunch in the pub was routine, and sometimes required. By the end of the 80s, that was at best frowned on. Over the last 20 years, having alcohol on your breath after lunch would have been a disciplinary issue, unless you were entertaining a client, at least in the places I worked.

Re: What every software developer must know about Unicode in 2023

#488

Post by extraduder_ire » Tue, Oct 03, 2023, 1:21 PM UTC

riggsdk wrote: post #374

posix86 wrote: post #329

Earlier quoted context omitted.

But typing "ö" (e.g. swiss keyboard) and pressing delete & getting an o would be annoying af

I realize that the editor would be the system to keep track of how the character was entered for this to work. If you made the character from a single keypress it would only make sense that backspace also undid the entire character. Only if you created the character from multiple keypresses it would make sense to "undo" only part of it with backspace (at least until you move away from the character).

The text already contains the data about how the character was entered (more specifically, what codepoints it's made of), unless it has been normalized somehow. It seems like a tarpit to me to properly specify these behaviours in a way that won't be an annoyance/surprise to a lot of people.

Re: What every software developer must know about Unicode in 2023

#489

Post by extraduder_ire » Tue, Oct 03, 2023, 1:26 PM UTC

Etherlord87 wrote: post #459

haberman wrote: post #364

Earlier quoted context omitted.

In that case, it sounds like `length` on Unicode strings simply shouldn't exist, since there is no obvious right answer for it. Instead there should be `codepointCount`, `graphemeCount`, etc.

You're absolutely correct! `length` is ambiguous - you shouldn't have a `time` argument in a `sleep` function; you should have `milliseconds` and/or `seconds` etc.

You could have a Duration argument though.

The parallels of string length with the phrase "How long is a piece of string?"[0] make this apparent/amusing. I'm sure I'm not the first person to think that.

[0]: https://en.wiktionary.org/wiki/how_long_is_a_piece_of_string

Re: What every software developer must know about Unicode in 2023

#490

Post by extraduder_ire » Tue, Oct 03, 2023, 1:40 PM UTC

aragonite wrote: post #35

davidham wrote: post #10

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

I've been drawing circles for over a minute now and no one has joined me yet, so I conclude those movements are random rather than made by intelligent beings. :)

I did the same for a while while I was reading. From another comment, the position only seems to update once a second, so it'll be hard for someone to notice your movements.