Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

101–110 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#101

Post by ars »

Arnavion wrote: post #92
ars wrote: post #16

Am I wrong for assuming the .length should return a length in bytes? If you want to use 32bit units, then multiply your output by 4. If you want to do Unicode string manipulation and length counting, then use specific functions for that - but the base internal .length function should just output bytes.

>then multiply your output by 4. That is not how UTF-32 works. >but the base internal .length function should just output bytes. Do you think the length of an `int64_t[3]` array should be 3 or 24?

> Do you think the length of an `int64_t[3]` array should be 3 or 24?

There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes.

In this example, the base function should do bytes, and there should be a unicode function to count it in other ways.

I could be sizing to fit in a database, or send over the wire, or I might want visible space on the screen, or I might want to know how to move the cursor.

Each of those types of length should be supported.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#103

Post by masklinn »

tehsauce wrote: post #20
benatkin wrote: post #8

Earlier quoted context omitted.

I think he meant to bring out defensiveness with that quip. He never says that it's a big deal, just that it's the worst way to get the length of a string containing emoji, presumably of the mainstream languages.

Why is it the worst way though?

Because it’s never actually useful.

You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text.

A length in codepoints gives you nothing that’s really actionable, at least not that you’d need outside of a context where you could easily obtain it otherwise.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#104

Post by oaiey »

arcticbull wrote: post #24
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

I wanted to brainfart about that length in the typical assumed usage should be 1 ignoring the inner encoding of Unicode of emoji ... But your comment was spot on and showed me my own assumption would fall into exactly this view scheme.

Thank you. Have my upvote.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#105

Post by adastra22 »

Mawr wrote: post #87
lawn wrote: post #85

Earlier quoted context omitted.

Nah, that's just dumb. Rust's way of all strings being utf-8 and providing the different lengths depending on your needs is far superior. If you want something else than utf-8 you can use another data type, like a vector of bytes.

According to the article, Rust does the same thing - " ".len() == 17.

"".chars().count() == 5

Rust gives you the freedom to specify what you mean.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#106

Post by masklinn »

xeeeeeeeeeeenu wrote: post #34

I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are…

> UTF-32 code units are Unicode code points.

They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid.

> It is wrong to present them as equally arbitrary concepts.

Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#107

Post by baq »

Jach wrote: post #80

Unsurprising that (at least some implementation of) Swift does the least wrong thing in returning 1. I think it's also one of the few languages that will return a count of 1 for the madness that is country flag emojis https://docs.swift.org/swift-book/documentation/the-swift-pr...

'least wrong length method' for strings is not providing one in the first place.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#108

Post by baq »

jlebar wrote: post #48

I was 100% prepared to believe that the length of empty string in js is 7. Then upon opening the post I was 100% ready to believe that js has three different string length functions that all handle Unicode differently.

and the gist of the problem is that it doesn't.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#109

Post by jillesvangurp »

foxes wrote: post #77

All these abominations are because of non strict typing String = List ( Char ) Chars don’t have a length, like a number doesn’t have a length - unless you talk about number of bits. If you are working with strings stick with strings. The string of a single character should be “1”. Just enforce proper typing. Anything else is not consistent.

No, it's caused by cost. For example Java has a char type. It's a 16 bit numeric value because Java uses UTF-16 internally for encoding strings. Java Strings are basically immutable char arrays with some fluff around them. If you ask for the String length, it returns the length of the underlying array. Nice and simple and unsurprising. And relatively cheap. Most more recent languages use 8 bit bytes and UTF-8 instead because that is emerged as the most common character encoding. But UTF-16 was a reasonable choice a quarter century ago and the practical difference doesn't matter that much and changing it would be disruptive.

If you put unicode characters consisting of multiple data points into a String, it necessarily increases the amount of chars. There's no way around that. Because there is no such thing as a UnicodeChar type in Java. You can't actually assign multi data point unicode characters to a char.

Essentially all the workarounds for a 'correct' unicode character count in a String would either end up using a different and probably way more expensive data structure (e.g. a list of a list of chars or bytes where each list is a unicode character) or implementing some expensive logic for counting characters that is O(n) instead O(1). Most languages ranging from extremely strictly typed to weakly typed don't do that for cost reasons. The tradeoff is simply not worth the price it takes.

This stackoverflow post provides a few suggestions for how you could count 'correctly'. https://stackoverflow.com/questions/15947992/java-unicode-st... that illustrates the point nicely.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#110

Post by masklinn »

mjevans wrote: post #100
Dylan16807 wrote: post #73

Earlier quoted context omitted.

Lots of systems use UTF-16 internally and externally. Counting bytes in UTF-16 is, on average, almost as useful as counting bytes in UTF-8. I don't think just about anything communicates in UTF-32. 5 is basically just a codepoint count, and as such I don't think its usefulness rating should be between the byte counts.

Only Windows and Java come to mind - and BOTH of those are insane for sticking to it when the entire rest of the world has moved on.

Windows, Java, C#, javascript, a surprising number of XML documents (though less so as time marches on thankfully), ICU I think uses UTF-16 internally (for the same historical reasons as the other 4), JOLIET file names are UCS2, some phones interpret “16-bit” SMS as UTF-16 (the spec says UCS2).

> and BOTH of those are insane for sticking to it

They don’t really have much of a choice because they exposed those semantics as part of the string interface (or for Windows the interaction is slow low level it can’t be hidden), they have performance guarantees and behaviours which matches that.

It’s also why Python uses UTF-32, and went through the entire PEP-393 / FS complication to try and stop blowing up memory left and right: the core team considered that switching strings to UTF8 was a bridge too far.

There are approximate solutions, but they come with their own costs and complications (e.g. pypy uses UTF8 strings with lazily constructed indices to emulate UTF-32 strings).

Post reply on HN