61–70 of 121 posts
Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.
In practice, UTF-16 validation is rare.
Earlier quoted context omitted.
Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words). It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the…
Why is that a problem? That's how unix does it as well. A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.
That works fine until it becomes literally impossible when your language/library/framework forces you into picking an encoding.
Even Microsoft's own APIs that add "UTF-8 support" choke when they encounter file names with invalid UTF-16 sequences.
Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "…
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…
Earlier quoted context omitted.
The problem is that if the operating system allows sequences that are not valid UTF-16, then you cannot just state "it uses UTF-16": anything that is expecting UTF-16 and only UTF-16 is going to break on valid file names. "That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. I…
Does this actually matter? Who is out there, in the wild, creating file paths that use incredibly cursed random sequences of bytes? Maybe it's just me, but I don't really see the need to accommodate people who do stupid things to see what breaks.
On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only support UTF-8 paths.
Earlier quoted context omitted.
You’re definitely wrong. You’re designing a language that only works for “Americans” by default. Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English? You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American. And even if you’re American, people will still throw emdashes and Unicode quote symbols…
I mean the reserved words in my lang are in English, and the user-created identifiers must be ASCII at the moment. I could extend the latter to some wider range but my question was more about providing built-in Unicode string handling. My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes. I get your concern though and realize parsing let foo = " " would be a problem.
Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively.
Imagine being from a country where a language’s built in Strcmp doesn’t work?
TIL of UTF-1, what an odd specification.
Looks like the entire idea is to avoid using any ASCII/Extended ASCII control character bytes, except to represent those characters.
Not sure what the advantage of that was.
Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?
One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "cha…
Yes.
Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as
s(5)Hello
Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check.
Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as
(SMILING FACE)(RED HEART)
Earlier quoted context omitted.
Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surroga...
Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words). It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the…
You can't just dismiss the whole operating system API space just because NTFS doesn't enforce filename encoding. Yes, you might encounter file names with invalid UTF sequences because some app might be allowing them to be created. That can also happen in Linux too. Then add Linux, and probably others to the list. Then, what's the use for this list?
Earlier quoted context omitted.
Does this actually matter? Who is out there, in the wild, creating file paths that use incredibly cursed random sequences of bytes? Maybe it's just me, but I don't really see the need to accommodate people who do stupid things to see what breaks.
On Windows, it doesn’t matter much because, although the file system doesn’t validate Unicode, the usual system calls that work with file names do , so it’s very rare to encounter non-Unicode file names. On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only suppo…
I'm getting strong XKCD "Workflow"[0] vibes out of the idea that someone would have unintelligible paths on their system
Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "…
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…