Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

161–170 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#161

Post by wizofaus »

Mawr wrote: post #88
eviks wrote: post #65

Earlier quoted context omitted.

Why would you want dumb??? (and it's not expected that a character's length is>1 unless you've been conditioned to excpect it)

Because whenever you want to store or transmit a string only the byte count matters (the size of the string). All the fancy unicode stuff on top of bytes is for the display layers to handle. The default should be grounded to the reality of the programmer.

Storing and transmitting is always going to work with low-level storage units like bytes, so your string will need to be converted to that first. But string manipulation is extremely common in programming, and I would think graphemes are the most useful unit here - i.e. as a programmer my preference would be for swift's behaviour.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#162

Post by tremon »

amake wrote: post #152
ant6n wrote: post #90

Earlier quoted context omitted.

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accompli…

It's ambiguous because it's not clear what elements go into separate cells of the array.

Why is it ambiguous? The Python documentation is pretty clear about what type of elements a string contains:

> Strings are immutable sequences of Unicode code points

(from https://docs.python.org/3/library/stdtypes.html#text-sequenc...)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#163

Post by xeeeeeeeeeeenu »

masklinn wrote: post #106
xeeeeeeeeeeenu wrote: post #34

I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are…

> UTF-32 code units are Unicode code points. They’re not. UTF-32 code units have a 1:1 mapping to USVs , surrogates are not valid. > It is wrong to present them as equally arbitrary concepts. Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.

>They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid.

This is true, although very pedantic and irrelevant to the point of my comment. The distinction only matters when you're dealing with ill-formed strings.

BTW, Python strings can store surrogates.

>Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.

If you aren't directly dealing with UTF-16, UTF-16 code units aren't useful at all.

Code points/USVs, OTOH, are the building blocks of Unicode strings and various Unicode algorithms operate on them. They're low-level, but not useless.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#164

Post by lmm »

chasontherobot wrote: post #95
lmm wrote: post #43

Earlier quoted context omitted.

The argument is that indexing by codepoint is even less useful than indexing by byte.

As someone who has done both, I'd say that argument is wrong. It is much more convenient to index by code point. Indexing by bytes is almost always what you don't want to do, and leads to a lot of errors.

What were the use cases where you found it useful to index by code point (and therefore not by grapheme cluster)?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#165

Post by Findecanor »

ceeam wrote: post #71

These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.

I agree that they introduced unnecessary complexity for text encoding, and for font-rendering (which are expected to support multi-coloured emoticons now). I once started writing on a text editor, and then fell deep into Unicode handling. I have now spent more work on the Unicode parts than on anything else in the program.

I think that the industry could have instead adopted the old web-forum convention of colon-word-encoding, originating from ASCII art. Example: ":facepalm:". When the sequence is not supported as an emoji, it degrades gracefully into text that can be understood by anyone reading it instead of into a sequence of empty squares or diamonds with question marks in them. Text also provides a more efficient input method than having to browse for an icon in a list.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#166

Post by lmm »

MobiusHorizons wrote: post #51
lmm wrote: post #42

Earlier quoted context omitted.

> That means 7 is also a measure of bytes, just slightly more awkward. It's not a real measure of bytes though. It's the count of bytes in an encoding scheme that is (probably) neither what you use to communicate with the outside world nor what your language runtime uses. (And certainly it's no better than 5, since that's also a measure of bytes in a particular encoding).

The JavaScript language forces utf16 (whether or not v8 uses that representation under the hood). For instance if you want to substring the indexes you pass are for utf16 codepoints

Sure, but arguing that that's a good reason for length to count utf16 is purely circular.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#167

Post by lmm »

Dylan16807 wrote: post #73
lmm wrote: post #42

Earlier quoted context omitted.

> That means 7 is also a measure of bytes, just slightly more awkward. It's not a real measure of bytes though. It's the count of bytes in an encoding scheme that is (probably) neither what you use to communicate with the outside world nor what your language runtime uses. (And certainly it's no better than 5, since that's also a measure of bytes in a particular encoding).

Lots of systems use UTF-16 internally and externally. Counting bytes in UTF-16 is, on average, almost as useful as counting bytes in UTF-8. I don't think just about anything communicates in UTF-32. 5 is basically just a codepoint count, and as such I don't think its usefulness rating should be between the byte counts.

> Lots of systems use UTF-16 internally and externally. Counting bytes in UTF-16 is, on average, almost as useful as counting bytes in UTF-8.

Not my experience at all. The article points out that even languages that are committed to an UTF-16 interface prefer to use other internal storage representations, and I can't remember the last time I saw it used in a transfer format.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#168

Post by ant6n »

coldtea wrote: post #115
ant6n wrote: post #90

Earlier quoted context omitted.

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accompli…

> length is not ambiguous at all. Its the number of elements in the array That's because you defined it first as "the number of elements in the array". It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave. Not to mention there might not be any "array". A string (whatever the encoding / representation) is a c…

The python doc says "str" are immutable sequences of unicode code points. Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing). I couldn't find out in the documentation whether the __getitem__ is O(1), which I consider a deficiency -- this should definitely be well documented.

It doesn't really matter how some people think "how people understand" something, the documentation matters. Any string in any language is some ordered sequence of atomic text-like objects, so python's approach isn't unreasonable or unexpected, either.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#169

Post by pavlov »

eviks wrote: post #55

Very good and informative article, though still not convincing that the nudge to make the shortest "len" command use the human readable size of grapheme clusters like in Swift isn't the best design approach, all the non-intuitive sizes should be special

The article shows that the Swift approach produces different values for length depending on operating system and text library versions. Is that really intuitive?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#170

Post by ant6n »

tsimionescu wrote: post #146
ant6n wrote: post #90

Earlier quoted context omitted.

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accompli…

Interestingly, the number of Unicode codepoints is probably the only measure of a string that is unlikely to ever be relevant to anyone in practice except when it happens to coincide with a different measure. It can't be used to determine length in bytes (important for storage or network transmission), it can't be used to determine number of displayed characters, it can't be used to safely split a string at some posi…

The length of an array should correspond to the number of elements. Since each element is a code point, it's the most relevant number if you intend to operate on individual elements. That is, the maximum index corresponds to the length of the array.

If you care about the number of bytes, or to operate on individual bytes, then convert to utf-8,16 or 32, and operate on the bytes object. If you wish to operate on grapheme clusters, then you could probably find some 3rd party Python library that allows you to represent and operate on strings in terms of grapheme clusters.

Post reply on HN