Pretty clearly, "every software developer" doesn't need to understand Unicode with this level of familiarity, much like "every programmer" doesn't need to know the full contents of the 114 page Drepper paper. For example, I work on a GUID-addressed object store. Everything is in term of bytes and 128-bit UUIDs. Unicode is irrelevant to everyone on my team, and most adjacent teams. There is lots of software like this.
There seems to be quite large segments of developers working on functionality which "handles text" as immutable whole blobs, in which case one really doesn't have to know anything about unicode. However, as soon as you want to look into the text contents of the objects of that object store and handle parts of it, even in the very simplest way (e.g. checking whether the stored object contains some character, or whethe…
What every software developer must know about Unicode in 2023
401–410 of 572 posts
Re: What every software developer must know about Unicode in 2023
#402> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to? Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11. Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last…
I definitely do. The biggest difference, as everyone else, has pointed out is the US vs UK spellings.
Realistically, though, beyond that country is a poor indicator for everything else. I want to use DD/MM/YYYY date format in English, but DD.MM.YYYY date format in German. I want to use $1,000 in English, but 1.000 $ in German. This isn't dependent on the country I live in, this is dependant on a combination of a country and language - that could be the country I'm living in, or the country I grew up in (mostly US date format vs not), and it's either the language I'm actively typing in, or the language of the document I'm reading, or the language I'm thinking in (but a computer can't exactly handle that).
Trying to guess the correct combination is tricky, especially if a document is in two languages (e.g., a quotation), and users are lazy and won't switch their IME unless they have to.
What this means in slightly more practical terms is that setting a single "locale" for my device doesn't make sense, but rather I should be able to choose a locale per language (or possibly spelling preferences by language and formatting options by language as separate choices). I'd then pick a language to use the device in, and it would use that languages locale, and tell apps that this language is the preferred language. If an app doesn't provide my preferred language, pull the preferred locale from settings for a language it does support, otherwise use the default set by developers. For some apps, it's a bit more complex, particularly if I'm creating content. GMail or Office would be two good examples, where the UI language might be in English, but the emails or documents are in German, or a combination of German and English.
Even then, I'm sure there are people who need something even more flexible than that.
At the moment, if I set my language to English but my Country to Germany on my iPhone, for example, things occasionally get confused. My UK banking app, for example, pulled the decimal separator from my locale settings for a while and then refused to work because "£9,79" (or whatever it was) isn't a valid amount of money, and I couldn't see a way to fix that without switching my Country in the phone settings. I imagine they fixed it by ignoring my configured locale and always using en-GB, thus defeating the whole point of a locale in the first place.
So yeah, these days it's fairly common to not have a single "locale" that you work in - it's quite possible to want to use two or more but nothing is really set up to handle that well.
Re: What every software developer must know about Unicode in 2023
#403Earlier quoted context omitted.
> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".
In terminals there is a distinction between single-width and double-width characters (east-asian characters, in particular). E.g. the three characters A美C would take up the width of four ASCII monospace characters, the “美” being double-width. Similarly, for composed characters like say the ligature “ff”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).
Latin: Katakana
Full width: カタカナ
Half width: カタカナ
(How that fixed width text looks in a web browser is anyone’s guess though. On iOS none of the Japanese kana stay on the fixed grid.)Re: What every software developer must know about Unicode in 2023
#404Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen
Re: What every software developer must know about Unicode in 2023
#405> Unicode is a standard that aims to unify all human languages, both past and present, and make them work with computers. This is doubly wrong. First, it conflates languages and writing systems. Malay and English use the same writing system but are different languages. American Sign Language is a language, but it has no standard or widely-adopted writing system. Hakka is a language, but Hakka speakers normally write…
Re: What every software developer must know about Unicode in 2023
#406There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…
> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".
Re: What every software developer must know about Unicode in 2023
#407> For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points! It's a poor and misleading example for it is definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there (french is the language where 'é' is the most common). 'é' is U+00F9, one codepoint, definitely not two. Now you could say: but it…
I'm inclined to agree - they took a big problem and made it into a big & complex problem.
I propose a fix - immediately depreciate all multi-codepoint graphemes and provide single codepoint alternatives. That we should need to normalise something a basic as our text leaves soooo much room for problems. And the idea that some graphemes encode magically colour (emojis) ... :rolling_eyes_emoji:
Re: What every software developer must know about Unicode in 2023
#408Earlier quoted context omitted.
Thanks. Some alphabets have precomposed ligatures that aren't really letters, like old German alphabets with tz, ch, ss (I only know how to type the last one, ß, because the others have died out over the last hundred years). Actually in German (at least) ä, ö and ü really are actually ligatures for ae, oe, and ue -- the scribes started to write the E's on their sides above the base letters, and over time the superscr…
That's sounds a bit false to me. The Umlaute (ä,ö, ü) and the "eszett" ß are actually part of the German alphabet[1]. Also it is kinda weird to describe them as ligatures of the original letters and the diaeresis, because while this is what they started out as a long time ago, they are just their own letters now (as opposed to "real" stylistic ligatures like combining fi into one glyph). The advice your kid was told…
Also, though it's hardly authoritative, my kids' school taught English through immersion from grade 1 too, and both German and English teachers said "same alphabet".
As bmicraft pointed out, even in that wikipedia chart those inflected letters are spaced apart from the others. Yes, they are letterforms, but not part of the "alphabet" -- they don't even have a sorting like the Swedish Ä or W do.
And you can switch in running text from using the marker for umlaut (dots or bar, not semantically dieresis) or a normal "e" without anyone blinking. There's no problem reading a Swiss book even though ß refuses to cross the border. Though I personally prefer to read Äpfel and Bär rather than Aepfel and Baer, really, they are the same.
Re: What every software developer must know about Unicode in 2023
#409Earlier quoted context omitted.
> Yes, and that only makes things worse since the overwhelming majority of documents (99.something% last time I checked) uses pre-composed. It shouldn't matter what's in the wild in documents. That's why we have normalization algorithms and normalization forms. Unicode was built for the ugly reality of backwards compatibility and that you can't control how people in the past wrote. These precomposed characters largel…
The thing with normalization is that it's not free, and especially for embedded use cases people seem quite opposed to this. IIRC it requires about ~100K of binary size, ~20K of memory, and some non-zero number of CPU cycles. This is negligible for your desktop computer, but for embedded use cases this matters (or so I've been told). This comes up in specifications that have a broad range of use cases; when I was inv…
Sure, they want demonstrated usage as inline in the flow of text as textual elements as opposed to purely iconography or design elements (because such things are outside of Unicode's remit, modulo some old Wingdings encoded for compatibility reasons and the fine line between emoji are expressive text and also emoji are useful for iconography in many cases). But at this point (again in contrast to the UCS-2/no-Astral-plane days) the committees don't seem to care how it was mocked up (do it on a chalkboard, do it in paint, do it in LaTeX drawing commands, whatever gets the point across) or how "casual" or infrequent the usage is, so long as you can state the case for "this is a text element" (not an icon!) used in living creative language expression. There's more "provenance" requirements for dead languages and they'll want some number of academic citations, but for living languages they've come to be flexible (no hard requirements) on the number of examples they need from the wild and where those are sourced from. Showing it in old classic documents/manuals/books, for instance, helps the case greatly, but the committees today no longer seem as limited to just what can be used to demonstrate usage. "I just like it" is obviously not a rock solid proposal/defense to bring to a committee (any committee, really), but that doesn't mean that is impossible for the committee to be swayed by someone making a strong enough "I just like it" case if they demonstrate well enough why they like it and how they use it and how they think other people will use it (and how those uses aren't just iconography/decorative elements but useful in the inline context of textual language).
Re: What every software developer must know about Unicode in 2023
#410This is the most interesting comparison article I have seen in years about Unicode processing in C++: https://thephd.dev/the-c-c++-rust-string-text-encoding-api-l...
The author is also the lead on an open source C++ Unicode library called ztd.txt: https://github.com/soasis/text