Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

211–220 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#211

Post by insanitybit »

wtetzner wrote: post #199
zarzavat wrote: post #155

Earlier quoted context omitted.

There are three notions of length that make sense: 1. UTF-8 byte length 2. Code point count 3. Extended grapheme cluster count #3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level. I expect programming language string length to obey the law: len(a ++ b) = len(a) + len(b) For example, if I concatenate a two strings, one containing an “e” and one containing a…

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

I suspect primarily substring, where if you index by bytes you'll mangle the string, but if you index by codepoints everything works out.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#212

Post by insanitybit »

adastra22 wrote: post #105
Mawr wrote: post #87

Earlier quoted context omitted.

According to the article, Rust does the same thing - " ".len() == 17.

" ".chars().count() == 5 Rust gives you the freedom to specify what you mean.

Yeah but unfortunately it provides `.len()` directly. It's documented to make clear that it's the bytecount and not the characters, and that humans usually work with characters, but given that this isn't even a trait implementation I think `.as_bytes().len()` or something would have been better.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#213

Post by kps »

ceeam wrote: post #71

These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.

Unification was reasonable at the time, given the goal to fit Unicode in 16 bits, and willingness to exclude obsolete characters. It's just that they followed official Japanese standards, and therefore unified too many from the point of view of other languages.

I think the first big mistake was using postfix/infix operators (combining characters, modifiers, variant selectors, joiners, etc.) rather than prefix, preferably in blocks by arity. That would have simplified processing (in particular a keyboard dead key could have been identical to a combining character) and made broken sequences detectable.

The latest big mistake, I think, was retroactively changing some non-emoji characters to have “emoji presentation”, which means that some text has to be edited to preserve its original appearance.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#214

Post by speleding »

julik wrote: post #209

So many of these conversations could be easier if there would not be `length()` functions but `length_in_ ()` functions instead.

In ruby you have " ".codepoints.size == 5 and " ".bytes.size == 17

(It also has `length` which equals codepoints.size)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#215

Post by Symbiote »

rightbyte wrote: post #120
arcticbull wrote: post #24

Earlier quoted context omitted.

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> for western programmers And by western you mean american, right? You can't even use ASCII in the UK --- '£'.

For the UK, you also need to represent the Celtic languages. You'll need at least these letters: â, ê, î, ô, û, ŵ, ŷ, à, è, ì, ò, ù, ẁ (maybe ỳ?), á, é, í, ó, ú, ẃ (maybe ý?), ï (maybe more ¨), ...

Example: https://www.llyw.cymru/ (Welsh government homepage).

Bits of https://www.highland.gov.uk/press (Scottish Highlands council).

(I'll admit most people who don't speak these languages won't bother with the diacritics, unless it's as easy as typing English.)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#217

Post by Izkata »

arcticbull wrote: post #24
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> It is wrong that "{emoji}".length == 7

I get what you're intending, but it's kinda funny because that string actually is 7

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#218

Post by arp242 »

lmm wrote: post #164
chasontherobot wrote: post #95

Earlier quoted context omitted.

As someone who has done both, I'd say that argument is wrong. It is much more convenient to index by code point. Indexing by bytes is almost always what you don't want to do, and leads to a lot of errors.

What were the use cases where you found it useful to index by code point (and therefore not by grapheme cluster)?

In many cases it's not very useful, but there are clearly cases where it is, e.g. if you want to normalize text, compose/change emojis, stuff like that.

A codepoint is the "smallest useful addressable unit" when dealing with Unicode text, so it makes sense that's the default.

It's also comparatively expensive to address grapheme clusters.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#220

Post by blueflow »

zarzavat wrote: post #155
arcticbull wrote: post #24

Earlier quoted context omitted.

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

There are three notions of length that make sense: 1. UTF-8 byte length 2. Code point count 3. Extended grapheme cluster count #3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level. I expect programming language string length to obey the law: len(a ++ b) = len(a) + len(b) For example, if I concatenate a two strings, one containing an “e” and one containing a…

UCS-2 length is also something that is still required today.
Post reply on HN