291–300 of 315 posts
Obligatory: https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
Length = 5
Size: depends on the encoding
Width = 1
Earlier quoted context omitted.
As a hole , it would only be annoying and a performance penalty for validation. But by its very design, it will leak, and it does in such ways that it became the worst thing to ever happen to Unicode. I don’t know of a single language or library that uses UTF-16 for strings that validates strings: every last one actually uses sequences of UTF-16 code units, potentially ill-formed, and has APIs that guarantee this wil…
Let say A is an ill formed utf-16 string with unmatched surrogates. The problem comes when trying to convert A to utf-8. Is this the leak you are talking about?
The other is… well, much the same really, but when it makes it into specs that others have to care about. The web platform demonstrates this clearly: just about everything is defined with strings being sequences of UTF-16 code units (though increasingly new stuff uses UTF-8), so then other things wanting to integrate have to decide how to handle that, if their view of strings is different: whether to be lossy (decode/encode using REPLACEMENT CHARACTER substitution on error), or inconvenient (use a different, non-native string type). Rust has certainly been afflicted by this in a number of cases and ways, generally favouring correctness.
Earlier quoted context omitted.
Seeing a .length on something makes me think it’s an iterable without looking more deeply. In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once. ;)
Yeah; I've recently noticed that almost every time I use string.length in javascript, its wrong and going to break something as soon as emoji appears. In my code, I always want to deal with either the number of codepoints or the number of UTF8 bytes. String.length gives you neither, but unfortunately it looks correct until you test with non-ASCII strings.
Earlier quoted context omitted.
> Do you think the length of an `int64_t[3]` array should be 3 or 24? There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes. In this example, the base function should do bytes, and there should be a unicode function to count it in other ways. I could be sizing to fit in a database, or send over the wire, or I might want visible space on the screen, or I might want to know…
> There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes. That's not what that does, the answer to the code you wrote is 192 because you're multiplying the same size factor twice.
The first one has size 3, because there are 3 elements in the array. The second one has size 8 because an int64 is 8 bytes.
Earlier quoted context omitted.
Codepoints is best for collaborative text editing / CRDTs (diamond types, automerge, etc). We generally model the document as a big list of unicode codepoints. We could use grapheme clusters, but the grapheme cluster boundary points change as unicode evolves, and not all systems update at the same time. Separating strings based on grapheme cluster boundaries also requires a big lookup table to be embedded in every ap…
Using code points (or scalar values, I hope) just means that it’s inefficient for everyone , because now everyone has to convert indexes (well, except Python, but it has other problems), instead of only half the people. Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from dec…
The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because you need to reconstruct the whole document state to be able to tell if the edit is valid. Using Unicode codepoints makes invalid state unrepresentable. So the validation problem goes away. (You might still need to check that an insert isn’t past the end of the document, but that’s a much easier check).
Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) so you still need to convert positions anyway. Even in rust it’s common to see line/col positions from text editors.
Using utf8 byte offsets just doesn’t really give you any benefits in exchange for making validation much harder.
For all intents and purposes, a user will count it as one character. Truncating the string without including the whole cluster would change the meaning of it, and is not an operation anyone would do as a general purpose thing any more than someone would want to randomly replace the last character with random letters.
It looks like one character. I'd rather APIs let us continue pretending it is one character.
Earlier quoted context omitted.
Yeah; I've recently noticed that almost every time I use string.length in javascript, its wrong and going to break something as soon as emoji appears. In my code, I always want to deal with either the number of codepoints or the number of UTF8 bytes. String.length gives you neither, but unfortunately it looks correct until you test with non-ASCII strings.
At least the “of” loop returns what is selectable via a cursor in the browser. I think…
Earlier quoted context omitted.
> There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes. That's not what that does, the answer to the code you wrote is 192 because you're multiplying the same size factor twice.
No I'm not, read it again. The first one has size 3, because there are 3 elements in the array. The second one has size 8 because an int64 is 8 bytes.
You don't have to like it
Earlier quoted context omitted.
Using code points (or scalar values, I hope) just means that it’s inefficient for everyone , because now everyone has to convert indexes (well, except Python, but it has other problems), instead of only half the people. Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from dec…
No, I don’t agree. The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because yo…
> Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java)
Swift 5 switched to UTF-8: https://www.swift.org/blog/utf8-string/. I’m hopeful that other UTF-16 environments might eventually manage to switch to UTF-8 internally despite retaining some UTF-16 code unit semantics for compatibility; two projects have already demonstrated you can very practically do this sort of thing: Servo from fairly early on (WTF-8 despite the web’s UTF-16 code unit semantics), and PyPy since 7.1 (UTF-8 despite code point semantics, not sure what they do about surrogate code points). I know the web has largely backed away from UTF-16 and uses code point semantics (well, scalar values plus loose surrogates) on almost all new stuff, with good UTF-8 support too.