Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

111–120 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#111

Post by sirsinsalot »

ant6n wrote: post #90
jandrese wrote: post #38

Earlier quoted context omitted.

I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accompli…

Exactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a character and an array of runes could be encoded into bytes using a given encoding.

The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the length of an array of 2 runes was 4.

I always liked the rune unit, and while my memory is hazy I think it was just code points.

I think part of the issue is programmers and apis mixing bit units for in memory representation of a conceptual value mapping (unicode), conceptual characters, stored size when encoded and so on ... without firming up those abstractions with interfaces. It gets lossy.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#113

Post by masklinn »

BeefWellington wrote: post #33
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

5 makes perfect sense to me; the author's complaints seem kinda silly. An area this makes sense is, what do you expect to get if you do something like: emoji = " " print(emoji[:3]) Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes? Modern strings are complex objects that have evolved a bit past cha…

> Should this throw an error because there's only one displayed "character"?

Why should it not? You’re literally breaking the content.

Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue.

> Modern strings are complex objects that have evolved a bit past char[] or byte[].

And yet that’s exactly what you’re advocating, just with 21 bit chars.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#114

Post by frou_dh »

I encountered some real world unicode/emoji breakdown recently. I set my surname in a webapp to an emoji country flag because I needed a way to communicate where I was. Elsewhere in the app, it showed surnames as just their initial, e.g. "John S". There, mine showed as a featureless black flag rather than the flag I set. Presumably because that is the first codepoint of several that make up the flag.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#115

Post by coldtea »

ant6n wrote: post #90
jandrese wrote: post #38

Earlier quoted context omitted.

I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accompli…

>length is not ambiguous at all. Its the number of elements in the array

That's because you defined it first as "the number of elements in the array".

It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave.

Not to mention there might not be any "array". A string (whatever the encoding / representation) is a chunk of memory, not an array. That you can often use a method to traverse it doesn't mean it's in an array.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#116

Post by mjevans »

masklinn wrote: post #110
mjevans wrote: post #100

Earlier quoted context omitted.

Only Windows and Java come to mind - and BOTH of those are insane for sticking to it when the entire rest of the world has moved on.

Windows, Java, C#, javascript, a surprising number of XML documents (though less so as time marches on thankfully), ICU I think uses UTF-16 internally (for the same historical reasons as the other 4), JOLIET file names are UCS2, some phones interpret “16-bit” SMS as UTF-16 (the spec says UCS2). > and BOTH of those are insane for sticking to it They don’t really have much of a choice because they exposed those semanti…

I'm not a Windows based programmer, but couldn't they leave the old API's in place, but make UTF-8 safe versions available for everyone and switch to that... E.G. with Win 11?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#117

Post by tgv »

ars wrote: post #16

Am I wrong for assuming the .length should return a length in bytes? If you want to use 32bit units, then multiply your output by 4. If you want to do Unicode string manipulation and length counting, then use specific functions for that - but the base internal .length function should just output bytes.

The most obvious use case for length is iterating over the string and indexing it. In JS (or Go, Rust, Python) indexing and iteration is not byte based. As has been said elsewhere, length depends on the context/way you use it.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#118

Post by coldtea »

BeefWellington wrote: post #33
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

5 makes perfect sense to me; the author's complaints seem kinda silly. An area this makes sense is, what do you expect to get if you do something like: emoji = " " print(emoji[:3]) Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes? Modern strings are complex objects that have evolved a bit past cha…

>Should this throw an error because there's only one displayed "character"?

Absolutely.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#119

Post by coldtea »

emodendroket wrote: post #63
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

> I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them. Well, why not? There are a lot of things that people would want to call string.length for — drawing little equals signs under text in a terminal, for a frivolous…

>Well, why not?

Separation of concerns?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#120

Post by rightbyte »

arcticbull wrote: post #24
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> for western programmers

And by western you mean american, right? You can't even use ASCII in the UK --- '£'.

Post reply on HN