Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

221–230 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#221

Post by Izkata »

wtetzner wrote: post #199
zarzavat wrote: post #155

Earlier quoted context omitted.

There are three notions of length that make sense: 1. UTF-8 byte length 2. Code point count 3. Extended grapheme cluster count #3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level. I expect programming language string length to obey the law: len(a ++ b) = len(a) + len(b) For example, if I concatenate a two strings, one containing an “e” and one containing a…

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

Isn't that the level emoji sequences work at?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#222

Post by jakear »

dingledork69 wrote: post #67
jakear wrote: post #29

Earlier quoted context omitted.

Not sure if HN will handle it, but this empty string's length is 7: ' ' ( '\u200b'.repeat(7) ) Edit: HN does not, in fact, handle it.

HN does, in fact, handle it, and chose to discard it.

In the CIA since of "we need to handle these democratically elected leaders", perhaps.

But if I was selling you a drop-in comment widget and boasted "it handles all of Unicode", but really I was just running s/[^ -~]+/ /g, wouldn't you feel a bit let down?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#223

Post by tialaramex »

blueflow wrote: post #220
zarzavat wrote: post #155

Earlier quoted context omitted.

There are three notions of length that make sense: 1. UTF-8 byte length 2. Code point count 3. Extended grapheme cluster count #3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level. I expect programming language string length to obey the law: len(a ++ b) = len(a) + len(b) For example, if I concatenate a two strings, one containing an “e” and one containing a…

UCS-2 length is also something that is still required today.

UCS-2 is dead. You can't express Unicode in UCS-2. If you have old UCS-2 data you can just treat it as UTF-16, maybe check for encoding irregularities but if it was really UCS-2 correct Unicode it'll be fine.

UTF-16 length is only useful if you are moving UTF-16, perhaps for interop with other software that chose UTF-16. Remember to pass on your condolences and look forward to a day when we don't do that any more.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#224

Post by blueflow »

tialaramex wrote: post #223
blueflow wrote: post #220

Earlier quoted context omitted.

UCS-2 length is also something that is still required today.

UCS-2 is dead. You can't express Unicode in UCS-2. If you have old UCS-2 data you can just treat it as UTF-16, maybe check for encoding irregularities but if it was really UCS-2 correct Unicode it'll be fine. UTF-16 length is only useful if you are moving UTF-16, perhaps for interop with other software that chose UTF-16. Remember to pass on your condolences and look forward to a day when we don't do that any more.

UCS-2 support is shipped with every modern x86 firmware. UEFI system partitions use it for VFAT name encoding.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#225

Post by hateful »

The wife and I have a Google sheet that we use for our shared calendar - and we put an emoji before each "event" and in top row of each day I show the Emoji for that days entries. But I need to do:

> =LEFT(F280,2) & LEFT(F281,2) & LEFT(F282,2) & LEFT(F283,2)

Since the Emojis are actually 2 bytes.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#226

Post by vlmutolo »

PhilipRoman wrote: post #186

I cannot think of a single common case where grapheme cluster count is important. If you want to print them aligned to a terminal - guess what, double width characters exist, so the only reliable way is to print them first, measure the cursor movement using escape sequences, calculate length and erase the originally printed data. Even for limiting input field sizes byte count is much better, as otherwise you are open…

I think tweets are restricted by number of extended grapheme clusters. And probably also by number of bytes.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#227

Post by globular-toast »

dahfizz wrote: post #26

Maybe not wrong, but it's the worst option. 5 is the number of code points, and 17 is the number of bytes. Both are reasonable answers. 7 is the number of code units for utf-16. Seems like the least useful option.

Also it's trivial to get the number of bytes in Python if that's what is wanted:

    len(bytes(" ", "utf8")) == 17
This is really the only sane way and makes it explicit which encoding you are using.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#228

Post by [deleted] »

insanitybit wrote: post #211
wtetzner wrote: post #199

Earlier quoted context omitted.

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

I suspect primarily substring, where if you index by bytes you'll mangle the string, but if you index by codepoints everything works out.

[deleted]

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#229

Post by dveeden2 »

MySQL has two different length functions...

  mysql> WITH chars AS (SELECT ' ' c) 
      -> SELECT LENGTH(c), CHAR_LENGTH(c) FROM chars;
  +-----------+----------------+
  | LENGTH(c) | CHAR_LENGTH(c) |
  +-----------+----------------+
  |        17 |              5 |
  +-----------+----------------+
  1 row in set (0.01 sec)
Note that the doesn't seem to render in preformatted text on HN.

This should be easier to reproduce:

  mysql> WITH chars AS (SELECT 0xF09FA4A6F09F8FBCE2808DE29982EFB88F c)
      -> SELECT CONVERT(c USING utf8mb4), LENGTH(c), CHAR_LENGTH(c) FROM chars;
  +--------------------------+-----------+----------------+
  | CONVERT(c USING utf8mb4) | LENGTH(c) | CHAR_LENGTH(c) |
  +--------------------------+-----------+----------------+
  |                        |        17 |             17 |
  +--------------------------+-----------+----------------+
  1 row in set (0.00 sec)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#230

Post by seagreen »

arcticbull wrote: post #24
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

Fantastic comment, thank you.

One question I have is which of these are fixed across Unicode standards.

(a) Byte count for storage and (b) code point count are definitely fixed, Unicode provides an explicit guarantee of the latter: https://www.unicode.org/policies/stability_policy.html

(c) Visual width will change depending on the system.

But what about (d) grapheme count? If I make a microblogging site which limits post length to 144 graphemes, can my database invariants break when I upgrade my version of Unicode?

Post reply on HN