Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

301–310 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#301

Post by tsimionescu »

ant6n wrote: post #170
tsimionescu wrote: post #146

Earlier quoted context omitted.

Interestingly, the number of Unicode codepoints is probably the only measure of a string that is unlikely to ever be relevant to anyone in practice except when it happens to coincide with a different measure. It can't be used to determine length in bytes (important for storage or network transmission), it can't be used to determine number of displayed characters, it can't be used to safely split a string at some posi…

The length of an array should correspond to the number of elements. Since each element is a code point, it's the most relevant number if you intend to operate on individual elements. That is, the maximum index corresponds to the length of the array. If you care about the number of bytes, or to operate on individual bytes, then convert to utf-8,16 or 32, and operate on the bytes object. If you wish to operate on graph…

A string is not an array, it is a chunk of text, for the vast majority of uses of strings. Exactly how that chunk of text is represented in memory and what API it should expose is the discussion we're having. My point is that it shouldn't be exposed as an array of codepoints, since array operations (lengths, indexing, taking a range) are not a very useful way of manipulating text; and even if we did expose them as an array, Unicode code points are definitely not a useful data structure for almost any purpose.

There are basically only two things that can be done with a Unicode codepoint: encode it in bytes for storage, or transform it to a glyph in a particular font or culture.

You can't even compare two sequences of Unicode codepoints for equality in many cases, since there are different ways to represent the same text with Unicode. For example the strings "thá" and "thá" are different in terms of codepoints, but most people would expect to find the second when typing in the first. Even worse, there are codepoints which are supposed to represent different characters, depending on the font being used / the locale of the display (the same Unicode codepoints are used to represent related Chinese, Japanese, or Korean characters, even when these characters are not identical between the three cultures).

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#302

Post by tsimionescu »

avgcorrection wrote: post #240
tsimionescu wrote: post #146

Earlier quoted context omitted.

Interestingly, the number of Unicode codepoints is probably the only measure of a string that is unlikely to ever be relevant to anyone in practice except when it happens to coincide with a different measure. It can't be used to determine length in bytes (important for storage or network transmission), it can't be used to determine number of displayed characters, it can't be used to safely split a string at some posi…

Splitting into ASCII-only and Unicode would be more of a regression than a progression. And yes, the “I’m not a native speaker” is a typical pre-emptive reply, as if it matters (neither am I—doesn’t mean anything by itself).

Let me give an example of why I don't think a single unified string API works. When doing (stringA == stringB) , what do you expect to get as a result? Do you expect it to tell if you the two strings represent the exact same codepoints, or do you expect it to tell you whether they represent the same Unicode grapheme clusters, as Unicode recommends?

The answer is of course both, depending on context. You certainly don't want a fuzzy match when, say, decoding a protobuf, but you also don't want a codepoint match when looking up user input.

What most modern languages have settled on is having a Unicode codepoint array type, typically called string or text, and an array of bytes type. However, common string operations are often only provided for the text type, and not the bytes type - which becomes very annoying when doing low level work and using bytes for text, and hoping for simple text operations.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#303

Post by coldtea »

emodendroket wrote: post #282
coldtea wrote: post #119

Earlier quoted context omitted.

> Well, why not? Separation of concerns?

I’m not sure what separation of concerns is served by not providing a method to show how many glyphs are in a given string.

Who said there shouldn't be one? The point is there should be more than one, and that not all are a language/string library-level concern.

The context is "I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them."

This means that shouldn't be some generic "length" method, but appropriate separate-concerns methods (plural), some of which (e.g. regarding character width in pixels when rendered) even belong to a drawing lib and not the language at all.

The parent's point is that length (bytes), characters (count), and glyphs (size, shape) are different concerns. The latter would concern a drawing lib or a renderer, but not be a core string method (which should concern itself with the abstract notion of characters and the concrete notion of bytes).

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#304

Post by nerdponx »

chrismorgan wrote: post #290
nerdponx wrote: post #125

Earlier quoted context omitted.

It's useful if you want array-like semantics (e.g. O(1) lookup) on Unicode text strings, because you have a fixed size for every codepoint, unliked UTF-8. Python for example uses it internally.

Except code point indexing simply isn’t useful. In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”

I think that's the issue here. People disagree on how useful or not useful it is. It's maybe not ideal, but I don't think it's anywhere near so bad as to be entirely not useful. Strings-are-sequences-of-bytes is worse in my opinion. Python literally used to have that. It was worse.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#305

Post by wtetzner »

zarzavat wrote: post #232
wtetzner wrote: post #199

Earlier quoted context omitted.

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

Everything that you can do to a Unicode string, except concatenation, is defined in terms of code points. Normalization, case transformations, collation, regexes, layout and rendering and encoding. For example, let’s say you want to define a “natural sort” order that sorts e.g. “A2” < “A10”. To do that you divide the string at boundaries between code points in ranges of each numeral type that you are supporting (e.g.…

Wouldn't layout/rendering etc. be done in terms of grapheme clusters?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#306

Post by wtetzner »

josephg wrote: post #296
chrismorgan wrote: post #287

Earlier quoted context omitted.

Using code points (or scalar values, I hope) just means that it’s inefficient for everyone , because now everyone has to convert indexes (well, except Python, but it has other problems), instead of only half the people. Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from dec…

No, I don’t agree. The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because yo…

> Using Unicode codepoints makes invalid state unrepresentable.

Is that true? Maybe it's not "invalid", but you might very well slice through the middle of a grapheme cluster.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#307

Post by ars »

tialaramex wrote: post #299
ars wrote: post #295

Earlier quoted context omitted.

No I'm not, read it again. The first one has size 3, because there are 3 elements in the array. The second one has size 8 because an int64 is 8 bytes.

I'm just reporting facts, C's sizeof operator measures bytes, so sizeof(int64_t[3]) * sizeof(int64_t) is 192 You don't have to like it

I apologize, you are correct. It's been too long since I used c.

I should write instead: sizeof(int64_t[3]) / sizeof(int64_t) for length of array.

This does help my argument that Javascript length on a Unicode string should return bytes though :)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#308

Post by chrismorgan »

nerdponx wrote: post #304
chrismorgan wrote: post #290

Earlier quoted context omitted.

Except code point indexing simply isn’t useful. In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”

I think that's the issue here. People disagree on how useful or not useful it is. It's maybe not ideal, but I don't think it's anywhere near so bad as to be entirely not useful. Strings-are-sequences-of-bytes is worse in my opinion. Python literally used to have that. It was worse.

The problem with what Python used to have is that the encoding wasn’t fixed.

I’ll agree with you that strings-are-sequences-of-bytes is bad. That’s painful compiler-flag, codepage, &c. territory.

But what’s not bad is strings-are-sequences-of-code-units. That’s what Rust has, for example. Rust strings aren’t sequences of bytes, but of UTF-8 code units, and the two are semantically very different.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#309

Post by chrismorgan »

wtetzner wrote: post #306
josephg wrote: post #296

Earlier quoted context omitted.

No, I don’t agree. The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because yo…

> Using Unicode codepoints makes invalid state unrepresentable. Is that true? Maybe it's not "invalid", but you might very well slice through the middle of a grapheme cluster.

I contemplated querying that myself, but decided that for CRDT editing purposes, it’s probably never practical to think about grapheme clusters. Given text in a language with syllabic grapheme clusters (e.g. Indic languages), if you start with “ba” (one syllable/grapheme cluster) and edit it to “boca” (two syllables/grapheme clusters), you could say “replaced ‘ba’ with ‘boca’”, but I’d be surprised if any CRDT did it that way if it could instead handle it as “inserted ‘oc’”, even though “oc” mightn’t make sense by itself, linguistically. But Unicode doesn’t do much in the way of defining what makes sense or not, and I don’t think there’s any coherent “bad grapheme clustering” detection algorithm. (Aside: on reflection, my Indic languages example is messier still, since -a probably represents the inherent vowel, so in “ba” → “boca” those “a”s are probably actually represented by the absence of a vowel sign code point—and if you wanted to suppress the inherent vowel, you’d need to add a virama sign. Fun stuff.)

But then again, I know that some CRDTs struggle with interleaving, and maybe grapheme-awareness could help things out in some way or other. I dunno.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#310

Post by josephg »

chrismorgan wrote: post #309
wtetzner wrote: post #306

Earlier quoted context omitted.

> Using Unicode codepoints makes invalid state unrepresentable. Is that true? Maybe it's not "invalid", but you might very well slice through the middle of a grapheme cluster.

I contemplated querying that myself, but decided that for CRDT editing purposes, it’s probably never practical to think about grapheme clusters. Given text in a language with syllabic grapheme clusters (e.g. Indic languages), if you start with “ba” (one syllable/grapheme cluster) and edit it to “boca” (two syllables/grapheme clusters), you could say “replaced ‘ba’ with ‘boca’”, but I’d be surprised if any CRDT did it…

Yeah I agree. I think its inevitable that collaboratively edited documents sometimes end up with grapheme clusters that are considered invalid by some peers, simply because different peers might be using different versions of unicode. If my phone supports the polar bear emoji and yours doesn't, you'll see weird stuff instead of a polar bear. There's no getting around that.

And yes, using unicode codepoints, buggy clients might insert extra unicode characters in the middle of a grapheme cluster. But ... Eh. Fine. I'm not super bothered by that from a data validation perspective.

Why don't I have the same attitude toward invalid UTF8? I mean, the CRDT could syncronize arbitrary arrays of bytes that by agreement contain valid UTF8, and treat it as user error in the same way if that happens? Two reasons. First, because some languages (eg rust) strictly enforce that all strings must contain valid UTF8. So you can't even make a document into a String if it has invalid UTF8. We'd need a try_ codepath, which makes the API worse. Secondly, languages like javascript which store strings using UTF16 don't have an equivalent encoding for invalid UTF8 bytes at all. Javascript would have to store the document internally in a byte array or something, and decode it to a string at the frontend. And thats complex and inefficient. That all sounds much worse to me than just treating the document as a sequence of arbitrary unicode codepoints - doing which guarantees correctness and we don't need any of that mess.

Post reply on HN