Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

231–240 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#231

Post by globular-toast »

grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

> The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file.

Almost. 17 is the number of bytes it occupies in memory. But you don't generally dump memory directly to disk or network. It happens to make sense (and it's convenient) for utf8 strings. But it's better to be explicit about that. Python is better. If you care about bytes, say you care about bytes:

    len(bytes(" ", "utf8")) == 17
    len(bytes(" ", "utf-16-be")) == 14
    len(bytes(" ", "utf-32-be")) == 20

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#232

Post by zarzavat »

wtetzner wrote: post #199
zarzavat wrote: post #155

Earlier quoted context omitted.

There are three notions of length that make sense: 1. UTF-8 byte length 2. Code point count 3. Extended grapheme cluster count #3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level. I expect programming language string length to obey the law: len(a ++ b) = len(a) + len(b) For example, if I concatenate a two strings, one containing an “e” and one containing a…

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

Everything that you can do to a Unicode string, except concatenation, is defined in terms of code points. Normalization, case transformations, collation, regexes, layout and rendering and encoding.

For example, let’s say you want to define a “natural sort” order that sorts e.g. “A2” < “A10”. To do that you divide the string at boundaries between code points in ranges of each numeral type that you are supporting (e.g. western numerals, Arabic numerals, Chinese numerals).

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#233

Post by PhilipRoman »

zkldi wrote: post #208
PhilipRoman wrote: post #186

I cannot think of a single common case where grapheme cluster count is important. If you want to print them aligned to a terminal - guess what, double width characters exist, so the only reliable way is to print them first, measure the cursor movement using escape sequences, calculate length and erase the originally printed data. Even for limiting input field sizes byte count is much better, as otherwise you are open…

this doesn't "just work" at all? utf-8 is a variable width encoding and if you treat it like ascii your software just isn't going to work outside of the ascii range.

But it does - the genius of utf-8 is that it was deliberately designed to be backwards compatible (it even preserves the ascii sorting order). You can run C programs written before utf-8 was invented with utf-8 inputs (unlike with the abomination that is utf-16).

If a code point is outside the ascii range (0-127 inclusive), then it's utf-8 encoding is also guaranteed to not contain any ascii bytes. So as long as you treat anything inside 128-255 as "some unknown character", the utf-8 code points will be preserved and eventually displayed when the byte sequence is parsed as utf-8 by your terminal/browser/whatever.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#234

Post by tourist2d »

Izkata wrote: post #217
arcticbull wrote: post #24

Earlier quoted context omitted.

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> It is wrong that "{emoji}".length == 7 I get what you're intending, but it's kinda funny because that string actually is 7

The underlying size is 7, the length isn't 7.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#235

Post by josephg »

wtetzner wrote: post #199
zarzavat wrote: post #155

Earlier quoted context omitted.

There are three notions of length that make sense: 1. UTF-8 byte length 2. Code point count 3. Extended grapheme cluster count #3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level. I expect programming language string length to obey the law: len(a ++ b) = len(a) + len(b) For example, if I concatenate a two strings, one containing an “e” and one containing a…

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

Codepoints is best for collaborative text editing / CRDTs (diamond types, automerge, etc). We generally model the document as a big list of unicode codepoints.

We could use grapheme clusters, but the grapheme cluster boundary points change as unicode evolves, and not all systems update at the same time. Separating strings based on grapheme cluster boundaries also requires a big lookup table to be embedded in every app. Unicode codepoints are obvious, stable, and easy to work with. And they're encoding-agnostic, so there's no weird UCS2-to-UTF8-bytes conversion needed in javascript, C#, etc.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#236

Post by josephg »

sroussey wrote: post #54
jandrese wrote: post #38

Earlier quoted context omitted.

I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.

Seeing a .length on something makes me think it’s an iterable without looking more deeply. In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once. ;)

Yeah; I've recently noticed that almost every time I use string.length in javascript, its wrong and going to break something as soon as emoji appears. In my code, I always want to deal with either the number of codepoints or the number of UTF8 bytes. String.length gives you neither, but unfortunately it looks correct until you test with non-ASCII strings.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#237

Post by elcaro »

arcticbull wrote: post #24
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

This is exactly what Raku does. Neither strings nor arrays have a `.length` method because it's arbitrary.

    [0] > "\c[FACE PALM]".chars
    1
    [1] > "\c[FACE PALM]".codes
    1
    [2] > "\c[FACE PALM]".encode.bytes
    4
    [3] > "\c[FACE PALM]".encode
    utf8:0x
    [4] > "\c[FACE PALM]".encode('utf-16')
    utf16:0x
FWIW, to get the "length" of an array, the method is `.elems`.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#238

Post by globular-toast »

masklinn wrote: post #103
tehsauce wrote: post #20

Earlier quoted context omitted.

Why is it the worst way though?

Because it’s never actually useful. You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text. A length in codepoints gives you noth…

What is useful? What should it be? Don't say bytes because there is already an idiomatic way to get bytes: `len(bytes(s, enc))` which is both more correct and explicit.

Maybe it should just return None because the only useful thing is probably how much "space" it occupies on screen in a fixed-width font, but that's too difficult to know.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#239

Post by jlokier »

insanitybit wrote: post #211
wtetzner wrote: post #199

Earlier quoted context omitted.

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

I suspect primarily substring, where if you index by bytes you'll mangle the string, but if you index by codepoints everything works out.

You'll mangle the string if you index and search by code points too, when the string contains the emoji this article is about, or for that matter an "e" following by combining acute accent.

The string will be fine if you only move to, split and concatenate at indices outside those grapheme clusters. But that is also true when indexing by bytes or UTF-16 code units.

So in some senses, indexing by bytes is just as good as indexing by code points, but faster. Either way to avoid mangling strings you need to restrict the indices of whatever type to meaningful character boundaries.

If you have decided to avoid string indices inside grapheme clusters, there comes the awkard question of what should you do when editing text in an environment rendered with font ligatures like "->" rendered as → (rightward arrow). From one perspective, that's just a font. From another, the user sees a single character yet there are valid positions (such as from cursor movement and character search) that land mid-way through the character, and editing at those positions changes the character. Neither is clearly best for all situations.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#240

Post by avgcorrection »

tsimionescu wrote: post #146
ant6n wrote: post #90

Earlier quoted context omitted.

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accompli…

Interestingly, the number of Unicode codepoints is probably the only measure of a string that is unlikely to ever be relevant to anyone in practice except when it happens to coincide with a different measure. It can't be used to determine length in bytes (important for storage or network transmission), it can't be used to determine number of displayed characters, it can't be used to safely split a string at some posi…

Splitting into ASCII-only and Unicode would be more of a regression than a progression. And yes, the “I’m not a native speaker” is a typical pre-emptive reply, as if it matters (neither am I—doesn’t mean anything by itself).
Post reply on HN