It’s not wrong that "🤦🏼‍♂️".length == 7 (2019) • Hacker News Board

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#51

Post by MobiusHorizons » Fri, Jun 02, 2023, 2:35 AM UTC

lmm wrote: post #42

Dylan16807 wrote: post #30

Earlier quoted context omitted.

That means 7 is also a measure of bytes, just slightly more awkward. So it's roughly on par with 17. For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.

> That means 7 is also a measure of bytes, just slightly more awkward. It's not a real measure of bytes though. It's the count of bytes in an encoding scheme that is (probably) neither what you use to communicate with the outside world nor what your language runtime uses. (And certainly it's no better than 5, since that's also a measure of bytes in a particular encoding).

The JavaScript language forces utf16 (whether or not v8 uses that representation under the hood). For instance if you want to substring the indexes you pass are for utf16 codepoints

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#52

Post by arcticbull » Fri, Jun 02, 2023, 2:37 AM UTC

hgs3 wrote: post #50

arcticbull wrote: post #47

Earlier quoted context omitted.

There might be something a little imprecise here: code points vs code units vs character codes. I'm open to being wrong but I would be very surprised if they defined text as a "series of code units" the count of which can vary by encoding even for the same character. IMO in this context 'character codes' would likely be far more consistent with 'code points' and they're just trying to differentiate between styled and…

"Character code" is short for "character code point" or just code point. All Unicode algorithms and properties are defined in terms of the code point. UTF encodings are just a way of encoding a code point. From Unicode's perspective, you care about what is encoded (i.e. the code point) and not how it is encoded (i.e. UTF-8). Unicode is one of the most poorly understood topics. I think the confusion stems from 1. most…

I agree with everything you said, I think I'm just trying to reconcile that with the top of thread saying python was the most correct because it was returning '7 code points' and that 'UTF-whatever is an implementation detail'

But 7 is not the number of code points/USVs - that's the number of UTF-16 code units. The string is 5 USVs. If UTF-whatever is an implementation detail, wouldn't the correct answer to length be 5?

What am I missing haha.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#53

Post by secret-noun » Fri, Jun 02, 2023, 2:38 AM UTC

Related: I wrote a little web app that lets you see the codepoints for text like this

https://unicode-x-ray.com/?t=%F0%9F%A4%A6%F0%9F%8F%BC%E2%80%... (sorry if link looks scary, that's just the URL encoding of this emoji)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#54

Post by sroussey » Fri, Jun 02, 2023, 2:40 AM UTC

jandrese wrote: post #38

arcticbull wrote: post #24

Earlier quoted context omitted.

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.

Seeing a .length on something makes me think it’s an iterable without looking more deeply.

In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once.

;)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#55

Post by eviks » Fri, Jun 02, 2023, 2:41 AM UTC

Very good and informative article, though still not convincing that the nudge to make the shortest "len" command use the human readable size of grapheme clusters like in Swift isn't the best design approach, all the non-intuitive sizes should be special

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#56

Post by omoikane » Fri, Jun 02, 2023, 2:43 AM UTC

Ruby gives you the choice to iterate over all types, via `each_byte`, `each_char`, `each_codepoint`, or `each_grapheme_cluster`.

https://ruby-doc.org/3.2.2/String.html#class-String-label-Me...

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#57

Post by hgs3 » Fri, Jun 02, 2023, 2:44 AM UTC

arcticbull wrote: post #52

hgs3 wrote: post #50

Earlier quoted context omitted.

"Character code" is short for "character code point" or just code point. All Unicode algorithms and properties are defined in terms of the code point. UTF encodings are just a way of encoding a code point. From Unicode's perspective, you care about what is encoded (i.e. the code point) and not how it is encoded (i.e. UTF-8). Unicode is one of the most poorly understood topics. I think the confusion stems from 1. most…

I agree with everything you said, I think I'm just trying to reconcile that with the top of thread saying python was the most correct because it was returning '7 code points' and that 'UTF-whatever is an implementation detail' But 7 is not the number of code points/USVs - that's the number of UTF-16 code units. The string is 5 USVs. If UTF-whatever is an implementation detail, wouldn't the correct answer to length be…

Python does return 5. JavaScript returns 7. Python is returning the number of code points, JavaScript is returning the number of UTF-16 code units.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#58

Post by arcticbull » Fri, Jun 02, 2023, 2:45 AM UTC

hgs3 wrote: post #57

arcticbull wrote: post #52

Earlier quoted context omitted.

I agree with everything you said, I think I'm just trying to reconcile that with the top of thread saying python was the most correct because it was returning '7 code points' and that 'UTF-whatever is an implementation detail' But 7 is not the number of code points/USVs - that's the number of UTF-16 code units. The string is 5 USVs. If UTF-whatever is an implementation detail, wouldn't the correct answer to length be…

Python does return 5. JavaScript returns 7. Python is returning the number of code points, JavaScript is returning the number of UTF-16 code units.

There's my mistake. Thank you. Flipped them in my head, it's ben a long day.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#59

Post by prng2021 » Fri, Jun 02, 2023, 2:46 AM UTC

arcticbull wrote: post #24

grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> A string should be viewed as an opaque data type with views into it depending on what you're trying to do.

> The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view.

That’s a great way to explain it and something I’ll keep with me. Thanks

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#60

Post by eviks » Fri, Jun 02, 2023, 2:46 AM UTC

arcticbull wrote: post #24

grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

What do you mean there is no such thing as a character when grapheme cluster is exactly that? This is also the out-of-context , and people get confused because instead of this human context attribute they've been forced to use all the other alternatives that require more knowledge