81–90 of 121 posts
A reviewer for a scientific paper I wrote asked if I had considered the impact of UTF-9 on what I was doing. To this day I don't know if they were trolling me.
Earlier quoted context omitted.
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…
The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?
Failing that, ask an AI.
There's no way to take an arbitrary utf-8 string and shorten it to Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker.
Strings are language. Language is hard.
Earlier quoted context omitted.
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…
How many bytes should be removed from your gap buffer when the user presses backspace? Does it matter if it's Hangul?
Earlier quoted context omitted.
How many bytes should be removed from your gap buffer when the user presses backspace? Does it matter if it's Hangul?
It's possible that pressing backspace needs to ADD bytes.
Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how people think of their characters.
Is there a real world case? I'd be excited to learn it.
Earlier quoted context omitted.
It's possible that pressing backspace needs to ADD bytes.
What is that case? Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how…
... Hangul.
What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)
Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "…
Earlier quoted context omitted.
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…
How many bytes should be removed from your gap buffer when the user presses backspace? Does it matter if it's Hangul?
The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.
Earlier quoted context omitted.
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…
The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?
Earlier quoted context omitted.
It's terrible, and we IMHO owe that to some introductory university courses to Java (plus some Win32 veterans). I got very close to being rejected by a professor that was obstinately convinced that Unicode "characters" were 2 bytes because it drunk the Kool Aid in the '90s about Java's `char` type representing a Unicode character. Ugh. I still get angry by thinking back at that sometimes
I can relate, I remember a teacher stating "you never have to worry about the amount of memory". This was in the late 90s, I then asked "So I can load a 20 gig data file into memory", he said yes.
Earlier quoted context omitted.
People expect Strings. You don't have to provide strings, but you should be aware that people expect them and if you provide something else but don't clearly label it as not strings there's an excellent chance they'll go "Oh, strings" and then they'll complain that it doesn't work the way they expected. If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody…
> If I can write "foo".contains('o') [..] but then you pretend you don't have strings, [..] C has strings By your logic, couldn't I pretend then to have strings insomuch as C when "foo".contains('o') can be transpiled by my lang as strchr("foo", 'o') != NULL