One of my favorite interview questions is simply "What's the difference between Unicode and utf-8?" I feel like that's pretty mandatory knowledge for any specialty but it doesn't get answered correctly very often.
What every software developer must know about Unicode in 2023
421–430 of 572 posts
Re: What every software developer must know about Unicode in 2023
#422There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…
> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".
﷽
𒐫
𒈙
꧅
Re: What every software developer must know about Unicode in 2023
#423> Unicode is a standard that aims to unify all human languages, both past and present, and make them work with computers. This is doubly wrong. First, it conflates languages and writing systems. Malay and English use the same writing system but are different languages. American Sign Language is a language, but it has no standard or widely-adopted writing system. Hakka is a language, but Hakka speakers normally write…
> Second, it's not that case that Unicode aims to encode all writing systems. For example, there are many hobbyist neographies (constructed writing systems) which will not be included in Unicode.
Doesn't the "private use space" technically satisfy this?
Re: What every software developer must know about Unicode in 2023
#424There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…
I'm not Korean but seeing that said of the Hangul example definitely made me pause - I doubt Koreans think of that example as a single grapheme (open to correction), though it is an excellent example all the same since it demonstrates the complexity of defining "units" consistently across language. It reminds me a little of Open Street Map's inconsistent administrative hierarchies ("states", "countries", "counties",…
As a native Korean, I can confirm that "각" is perceived as a single character. But the example itself is bad anyway because everyone use a precomposed form U+AC01 instead of U+1100 U+1161 U+11A8 instead (they are canonically equivalent). This is more clear when you also consider a compatibility representation "ㄱㅏㄱ" U+3131 U+314F U+3131, which is same to "각" after compatibility normalizations (NFKC or NFKD), but perceived as three atomic characters in general.
Re: What every software developer must know about Unicode in 2023
#425> The only modern language that gets it right is Swift: I disagree. What is the "right" things is use-case dependent. For UI it's glyph bases, kinda, more precise some good enough abstraction over render width. For which glyphs are not always good enough but also the best you can get without adding a ton of complexity. But for pretty much every other use-case you want storage byte size. I mean in the UI you care abou…
*nod*
Rust was given as one of the examples and Rust's .len() behaviour is chosen based on three very reasonable concerns:
1. They want the String type to be available to embedded use-cases, where it's not reasonable to require the embedding of the quite large unicode tables needed to identify grapheme boundaries. (String is defined in the `alloc` module, which you can use in addition to `core` if your target has a heap allocator. It's just re-exported via `std`.)
2. They have a policy of not baking stuff that is defined by politics/fiat (eg. unicode codepoint assignments) into stuff that requires a compiler update to change. (Which is also why the standard library has no timezone handling.)
3. People need a convenient way to know how much memory/disk space to allocate to store a string verbatim. (Rust's `String` is just a newtype wrapper around `Vec` with restricted construction and added helper functions.)
That's why .len() counts bytes in Rust.
Just like with timezone definitions, Rust has a de facto standard place to find a grapheme-wise iterator... the unicode-segmentation crate.
Re: What every software developer must know about Unicode in 2023
#426Prior to this article, I knew graphemes were a thing and that proper unicode software is supposed to count those instead of bytes or code points. I didn't know that unicode changes the definition of grapheme in backwards incompatible fashion annually, so software which works by grapheme count is probably inconsistent with other software using a different version of the standard anyway. I'm therefore going to continue…
> I didn't know that unicode changes the definition of grapheme in backwards incompatible fashion annually, so software which works by grapheme count is probably inconsistent with other software using a different version of the standard anyway.
This is EXACTLY why Rust's standard library is blind to graphemes. Support for the case where your company requires a specially certified toolchain that lags five years behind Rust upstream is an explicit goal that they address by breaking the stuff that changes quickly out into minimal crates that can be audited, updated at a quicker pace without requiring toolchain updates, and which have the option to continue to support older compiler versions indefinitely.
Re: What every software developer must know about Unicode in 2023
#427> normalization A quick war-story on this: We had a system which was taking web-user input for human names, and then some of it had to be sanitized for a crappy third-party system. However some of the names were getting mangled in unexpected ways. One of the (multiple) issues was that we were sometimes entirely dropping accented characters even when a good alternative existed. This occurred when we were getting "é" (…
Give https://manishearth.github.io/blog/2017/01/15/breaking-our-l... a read and, ideally, the other things it mentions like https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/ and https://manishearth.github.io/blog/2017/01/14/stop-ascribing....
(Among other things, it points out that doing that to non-Latin text is liable to change pronunciations and meanings in other languages. For example, some languages use diacritics for voiced/unvoiced indication where your "normalization" could do things like "tick→dick" or "did→tit".)
(Did you ever notice that? B/P, D/T, V/F, G/K, J/CH, and Z/S form voiced/unvoiced pairs that could have been indicated with a single letter and a diacritic. Same mouth behaviour. It's just a question of whether you engage your vocal cords.)
Re: What every software developer must know about Unicode in 2023
#428>3 Grapheme Cluster Boundaries >It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a use…
Oh my god, is there ever anything simple about unicode
Give the "Indic scripts" section of https://manishearth.github.io/blog/2017/01/15/breaking-our-l... a read.
TL;DR: Unicode is complicated because some non-Latin writing systems are complicated and those non-Latin writing systems account for over a quarter of the world's population. (They're either majority or present in India, Indonesia, Pakistan, Bangladesh, the Philippines, etc.)
Re: What every software developer must know about Unicode in 2023
#429[deleted]
Re: What every software developer must know about Unicode in 2023
#430With the benefit of hindsight, would we include the error detection bits of UTF8 if we could choose not to?
Yes. Give https://www.youtube.com/watch?v=_mZBa3sqTrI a watch... especially the "Oh my God! We've been hacked!" part at 36:20.
TL;DR: They had a transient glitch in their network switch and, because Windows uses UTF-16 when sending remote event logs over the wire, whenever it dropped a single byte, it had the effect of swapping the endianness of the messages, resulting in scary Chinese text in the logs.
You could get the same effect by naively applying byte-wise processing to UTF-16 or UTF-32, or having an off-by-one error.
UTF-8 is self-synchronizing so one-byte errors like that only lose you one character, rather than corrupting the entire stream going forward.