71–80 of 121 posts
My favorite, similar to the issue of case-folding, is the idea of something being a "letter" and "lowercase" but NOT "lowercase letter."
if you see URLs which project the microsoft editors choice of "single quote mark" into a triad of %xx; values, you've seen how people wind up in unexpected unicode/utf land, because of the text editor swapping ASCII for an uplift into either distorted iso-latin1, or unicode/utf8 depending.
Also a reason why I'm against Unicode identifier in source code. You got some source with characters like 'ö' in one variant and now your editor decides to use a different variant (e.g. compatibility character vs base character + combining decoration). And from my observation programming languages which claim to support Unicode tend to not make the effort to resolve those equivalents. At least for older versions of Go the specification clearly stated that they only compare Unicode code points. (don't know the current state)
Earlier quoted context omitted.
On Windows, it doesn’t matter much because, although the file system doesn’t validate Unicode, the usual system calls that work with file names do , so it’s very rare to encounter non-Unicode file names. On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only suppo…
Like what? I'm getting strong XKCD "Workflow"[0] vibes out of the idea that someone would have unintelligible paths on their system [0] https://xkcd.com/1172/
I have quite a number of times. There's no one "thing"; it's usually some legacy app that write these names. And when you get a program that refuses to open any non-UTF16 compliant file name, you'll have a time of grief.
Earlier quoted context omitted.
I mean the reserved words in my lang are in English, and the user-created identifiers must be ASCII at the moment. I could extend the latter to some wider range but my question was more about providing built-in Unicode string handling. My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes. I get your concern though and realize parsing let foo = " " would be a problem.
Apologies if I come off a bit strong on this one. I deal mostly with non “American” text so it’s a pain point. Unicode is so messy, trying to push handling it on to each individual programmer is not an ideal situation. Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively. Imagine being from a country where a language’s…
> Imagine being from a country where a language’s built in Strcmp doesn’t work?
Problem is it's seemingly a massive headache. Citing the article, "Unicode handling is always best left to a library".
I could bundle the recommended lib (ICU) into my lang's stdlib but it seems quite big...
Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "…
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…
I used to work on a platform at a large financial services firm; it was essentially complete ignorant of anything Unicode with respect to string handling, strings were null-terminated byte streams. The platform had CSV import capability for tabular data, and it had an integrated pivot table capability based on some widgets that had been grafted onto it. Some of the users in Hong Kong discovered that you could import…
So if you just move strings around and concatenate them, you don't even need to be aware of Unicode, even things like templating can work by just processing bytes. Of course, when you get to editing and rendering, you get all the complexity.
Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "…
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…
Does it matter if it's Hangul?
Earlier quoted context omitted.
One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "cha…
> separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Yes. Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JS…
Greetings, Professor Falken.
(SMILING FACE)(RED HEART)Favourite unicode fact: properly rendering unicode requires understanding of the current geopolitical situation (Depending on whom you accept as a country and whom you do not, two country-code-letters may or may not render as a flag. This changes sometimes in today's world.). https://esham.io/2014/06/unicode-flags
Imho, unicode should stay out of politics. Country flags, vaccine syringes and pregnant men should have nothing to do with how computers handle text and writing systems.
Earlier quoted context omitted.
Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words). It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the…
Windows used to use UCS-2 because UTF-16 hadn't been invented yet. Any Windows after Windows 2000 supports UTF-16. You can't just dismiss the whole operating system API space just because NTFS doesn't enforce filename encoding. Yes, you might encounter file names with invalid UTF sequences because some app might be allowing them to be created. That can also happen in Linux too. Then add Linux, and probably others to…