Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

41–50 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#41

Post by ladberg »

2h wrote: post #25

this one one of those things that people point to when comparing languages, but in reality rarely matters. with Go, you just get the number of bytes, which the the correct default thing to do: https://godocs.io/builtin#len if the language default was anything other than this, THAT WOULD BE WRONG and unexpected. I would prefer the default to be the dumb, fast thing. then if I want the slow, fancy thing, I can import s…

I think to some extent it depends on the language. In the article they talk about Swift's implementation, which by default does the slow, fancy thing (but makes it easy to do the dumb, fast thing). String manipulation in Swift is almost certainly going to be used for a GUI for end users of many possible languages / locales, so it makes sense to spend the extra cycles to get the fancy version by default. If it isn't the default then you'll end up with half the apps on the App Store displaying broken text on line breaks, ellipses, wrapping, etc. on their hand-rolled UI stack.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#42

Post by lmm »

Dylan16807 wrote: post #30
dahfizz wrote: post #26

Maybe not wrong, but it's the worst option. 5 is the number of code points, and 17 is the number of bytes. Both are reasonable answers. 7 is the number of code units for utf-16. Seems like the least useful option.

That means 7 is also a measure of bytes, just slightly more awkward. So it's roughly on par with 17. For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.

> That means 7 is also a measure of bytes, just slightly more awkward.

It's not a real measure of bytes though. It's the count of bytes in an encoding scheme that is (probably) neither what you use to communicate with the outside world nor what your language runtime uses. (And certainly it's no better than 5, since that's also a measure of bytes in a particular encoding).

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#43

Post by lmm »

arp242 wrote: post #23
paddw wrote: post #6

I think this is a really a naming convention issue. Len() is ambiguous, you really want either num_chars() or utfxx_len(). Of course, the issue of what counts as a character is confusing in its own right...

In Python len() on a bytes type gives you the number of bytes, and len() on a str type gives you the number of codepoints. I think that makes sense, as strings are only intended to deal with text, and you should never have to worry about byte indexing at all.

The argument is that indexing by codepoint is even less useful than indexing by byte.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#44

Post by croes »

pjscott wrote: post #18
hgs3 wrote: post #15

Python 3's approach is the most correct: Unicode defines text as a sequence of code points. UTF-whatever is an implementation detail.

Treating Unicode strings as a sequence of code points is a completely valid thing to do, but is usually not what you actually care about when dealing with text. Really, are code points any less of an implementation detail?

I think parent means most correct of the three given examples from Rust, JS and Python.

Especially because the article says that Python's take is the worst.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#45

Post by hgs3 »

arcticbull wrote: post #36
hgs3 wrote: post #15

Python 3's approach is the most correct: Unicode defines text as a sequence of code points. UTF-whatever is an implementation detail.

> Unicode defines text as a sequence of code points. Does it? Do you have a link? [edit] I looked up the spec and here is what it says. > The Unicode Standard does not define what is and is not a text element in different processes; instead, it defines elements called encoded characters. An encoded character is represented by a number from 0 to 10FFFF_16, called a code point. A text element, in turn, is represented b…

Review chapter 2.2 Unicode Design Principles in the Unicode Standard: "Plain text is a pure sequence of character codes; plain Unicode-encoded text is therefore a sequence of Unicode character codes."

Text elements are an abstract concept whose definition depends upon what is being processed. It might be a grapheme, it might be word, etc...

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#46

Post by skykooler »

dang wrote: post #37
lifthrasiir wrote: post #2

HN discards emojis in the title. The original emoji was https://emojipedia.org/man-facepalming-medium-light-skin-ton... which consists of 5 Unicode code points. Also please make sure to read the first heading after the title, which summarizes the whole point of this essay.

Ok, we've remojied the title above. Edit: there are always exceptions - https://news.ycombinator.com/item?id=34460417

Interestingly, Firefox on Wayland renders the emoji correctly in the tab title, but the window title renders it as two rectangles and the male symbol. I assume this must be some difference between system fonts vs Firefox's fonts.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#47

Post by arcticbull »

hgs3 wrote: post #45
arcticbull wrote: post #36

Earlier quoted context omitted.

> Unicode defines text as a sequence of code points. Does it? Do you have a link? [edit] I looked up the spec and here is what it says. > The Unicode Standard does not define what is and is not a text element in different processes; instead, it defines elements called encoded characters. An encoded character is represented by a number from 0 to 10FFFF_16, called a code point. A text element, in turn, is represented b…

Review chapter 2.2 Unicode Design Principles in the Unicode Standard: "Plain text is a pure sequence of character codes; plain Unicode-encoded text is therefore a sequence of Unicode character codes." Text elements are an abstract concept whose definition depends upon what is being processed. It might be a grapheme, it might be word, etc...

There might be something a little imprecise here: code points vs code units vs character codes.

I'm open to being wrong but I would be very surprised if they defined text as a "series of code units" the count of which can vary by encoding even for the same character. IMO in this context 'character codes' would likely be far more consistent with 'code points' and they're just trying to differentiate between styled and un-styled text. Whereas the 1.3 definition appears to be trying to make an authoritative definition of 'text.'

If we read 2.2's "character codes" as code points, then that can be multiple code points as referenced in 1.3

[edit] I originally flipped 'units' and 'codes' - cleaned it up.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#49

Post by MobiusHorizons »

dahfizz wrote: post #26

Maybe not wrong, but it's the worst option. 5 is the number of code points, and 17 is the number of bytes. Both are reasonable answers. 7 is the number of code units for utf-16. Seems like the least useful option.

It makes just as much sense as 17 (for utf8) in a JavaScript context, where charCodeAt(i) returns a utf-16 code point, and strings at least behave as though the implementation uses an array of uint16_t for the storage. Utf 16 is definitely not my favorite representation, but given that context (which the language imposes) 7 is an important number to be able to know.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#50

Post by hgs3 »

arcticbull wrote: post #47
hgs3 wrote: post #45

Earlier quoted context omitted.

Review chapter 2.2 Unicode Design Principles in the Unicode Standard: "Plain text is a pure sequence of character codes; plain Unicode-encoded text is therefore a sequence of Unicode character codes." Text elements are an abstract concept whose definition depends upon what is being processed. It might be a grapheme, it might be word, etc...

There might be something a little imprecise here: code points vs code units vs character codes. I'm open to being wrong but I would be very surprised if they defined text as a "series of code units" the count of which can vary by encoding even for the same character. IMO in this context 'character codes' would likely be far more consistent with 'code points' and they're just trying to differentiate between styled and…

"Character code" is short for "character code point" or just code point. All Unicode algorithms and properties are defined in terms of the code point. UTF encodings are just a way of encoding a code point. From Unicode's perspective, you care about what is encoded (i.e. the code point) and not how it is encoded (i.e. UTF-8).

Unicode is one of the most poorly understood topics. I think the confusion stems from 1. most programming languages getting the abstraction wrong, and 2. programmers trying to reconcile their non-technical interpretation of what "character" means.

Post reply on HN