Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

171–180 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#171

Post by eesmith »

sirsinsalot wrote: post #111
ant6n wrote: post #90

Earlier quoted context omitted.

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that. Remember, one of the big accompli…

Exactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a character and an array of runes could be encoded into bytes using a given encoding. The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the le…

FWIW, CPython uses one of several Unicode string implementation representations, depending on the code points involved:

  >>> import sys
  >>> s = 'A' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  1049
  >>> s = '\N{SNOWMAN WITHOUT SNOW}' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  2074
  >>> s = '\N{MUSICAL SYMBOL G CLEF}' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  4076
See https://peps.python.org/pep-0393/ . Mentioned in the linked-to article with "CPython since 3.3 makes the same idea three-level with code point semantics".

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#172

Post by lopis »

ceeam wrote: post #71

These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.

They were added to Unicode because they were already part of other encodings, and then were expanded. Makes total sense to add them.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#173

Post by tialaramex »

adastra22 wrote: post #158
tialaramex wrote: post #154

Earlier quoted context omitted.

Sure, however that's actually decoding the string into Unicode scalar values, and then counting them whereas the length of the string is a direct property of the string reference (it's a fat pointer [address + length]) I don't remember, but I think the size hint is set on the Chars iterator, so it can see it has 17 bytes of data, it knows that can't encode more than 17 Unicode scalar values, nor can it encode fewer t…

Yes, your point? That is the same thing which happens in Swift if you request the length of a string and it gives you the number of glyphs (1, in this case). Rust doesn't take sides here. It exposes all the different ways you might want to calculate the "length" of a string, and lets you pick which one you mean. The non-zero-cost choices involve a multi-step specification (like `.chars().count()`), which states expli…

Asking str.len() is a single very cheap operation, it's not only O(1) in the sense you'd learn in an algorithms course, it's really actually very cheap to do, it's fine if an algorithm relies heavily on str.len()

In contrast chars().count() creates an iterator and runs the iterator to completion counting steps, that's O(N) for a string of length N, and is in practice very expensive, you should definitely cache this value if you will need it repeatedly. It is possible the compiler can see what you're doing and cache it, but I am very far from certain so you should do so explicitly.

This is important in contrast to say, C, where strlen(str) is O(N) because it doesn't have fat pointers and so it has no idea how long the string is in any sense.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#174

Post by eviks »

pavlov wrote: post #169
eviks wrote: post #55

Very good and informative article, though still not convincing that the nudge to make the shortest "len" command use the human readable size of grapheme clusters like in Swift isn't the best design approach, all the non-intuitive sizes should be special

The article shows that the Swift approach produces different values for length depending on operating system and text library versions. Is that really intuitive?

Showing unintuitive results in a small number of cases always beats showing unintuitive results for all the cases.

The Swift approach can also asymptotically reach perfection. The other approach will be broken forever.

These are categorical improvements in design

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#175

Post by rightbyte »

mjburgess wrote: post #133
rightbyte wrote: post #120

Earlier quoted context omitted.

> for western programmers And by western you mean american, right? You can't even use ASCII in the UK --- '£'.

£ is still just a byte

'u32_pound & 0xff == u32_pound' happens to be true, ye. It doesn't make it a byte. You need the leading 0s.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#176

Post by ttlsa »

adastra22 wrote: post #105
Mawr wrote: post #87

Earlier quoted context omitted.

According to the article, Rust does the same thing - " ".len() == 17.

" ".chars().count() == 5 Rust gives you the freedom to specify what you mean.

len([]rune("")) does the same thing in Go.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#177

Post by ThunderSizzle »

tremon wrote: post #162
amake wrote: post #152

Earlier quoted context omitted.

It's ambiguous because it's not clear what elements go into separate cells of the array.

Why is it ambiguous? The Python documentation is pretty clear about what type of elements a string contains: > Strings are immutable sequences of Unicode code points (from https://docs.python.org/3/library/stdtypes.html#text-sequenc... )

According to python. Every language can have a different definition. C#, for example, defines the blocks to be Char objects, which is based on UTF 16:

> The Length property of a string represents the number of Char objects it contains, not the number of Unicode characters.

https://learn.microsoft.com/en-us/dotnet/csharp/programming-...

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#178

Post by thaumasiotes »

pajko wrote: post #129
arcticbull wrote: post #24

Earlier quoted context omitted.

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> Not until they have to explain to their designer why they can't limit a label to '10 characters.' Or in a single font. It's impossible to render any mixed combination of simplified Chinese, traditional Chinese and Japanese with a single font (Korean might be also involved, but not sure about that). Even in Unicode, characters might share the the same space which don't have anything common in their looks, nor in the…

> It's impossible to render any mixed combination of simplified Chinese, traditional Chinese and Japanese with a single font (Korean might be also involved, but not sure about that)

Well, that could be phrased better. Many such mixed combinations would encounter no problems. There is "Han Unification" in Unicode, in which certain graphical forms are declared equivalent and the intent is that they display as Japanese characters if you print them in a Japanese font, but as Chinese characters if you print them in a Chinese font. 直 is a good example of how that looks; try viewing it in different fonts.

But nobody likes unification and explicit fixed forms are constantly being defined so that it's possible to talk about them. Imagine if I wanted to write "in Old English, the word for dog was hund"... except that your font automatically replaced the sequence hund with a special ligature that looks exactly like dog.

So we have separate unicode points for ⻘ (modern, CJK RADICAL BLUE) and ⾭ (old, KANGXI RADICAL BLUE), and for ⿓ (traditional Chinese, KANGXI RADICAL DRAGON), ⻰ (simplified Chinese, CJK RADICAL C-SIMPLIFIED DRAGON), and ⻯ (Japanese, CJK RADICAL J-SIMPLIFIED DRAGON). Interestingly, the dragon characters are all considered different according to the original "Han Unified" specification, where they are CJK UNIFIED IDEOGRAPH 9F8D, CJK UNIFIED IDEOGRAPH 9F99, and CJK UNIFIED IDEOGRAPH 7ADC. In contrast, there is only the one "unified" form of 直, CJK UNIFIED IDEOGRAPH 76F4, but you can refer to its Chinese form explicitly with CJK COMPATIBILITY IDEOGRAPH FAA8 and to its Japanese form with CJK COMPATIBILITY IDEOGRAPH 2F940. (My browser font fails to render either of those.)

It was never possible to rely entirely on the font to handle dealing with simplified vs traditional characters for you, for the obvious reason that their mapping is not one-to-one. In simplified Chinese, 后 means "after"† or "behind" and it also means "empress". In traditional Chinese, "after" and "behind" would be 後. And "empress" would be... 后. This means there can be no way for a traditional Chinese font to determine what it should display if you write 后.

Ultratraditional Korean hanja participate in the same variation of forms that we see between Chinese and Japanese. But it isn't normal to write Korean in hanja outside of very specific contexts. Hangul are radically different and belong to a separate part of unicode entirely.

† "After" in time. "After" in sequence is 下, "below".

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#179

Post by pavlov »

eviks wrote: post #174
pavlov wrote: post #169

Earlier quoted context omitted.

The article shows that the Swift approach produces different values for length depending on operating system and text library versions. Is that really intuitive?

Showing unintuitive results in a small number of cases always beats showing unintuitive results for all the cases. The Swift approach can also asymptotically reach perfection. The other approach will be broken forever. These are categorical improvements in design

The Swift approach can't reach perfection in isolation because data from the future can always break it.

That's why in the article you see Swift running on Ubuntu 14.04 returning len==2 while the same code on Ubuntu 18.04 returns len==1 for the same emoji string.

IMO that's a big philosophical question here: do we accept that "string length" means something you can't compute for arbitrary strings unless your code is receiving annual updates containing the latest Unicode interpretation instructions?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#180

Post by rhaway84773 »

ant6n wrote: post #168
coldtea wrote: post #115

Earlier quoted context omitted.

> length is not ambiguous at all. Its the number of elements in the array That's because you defined it first as "the number of elements in the array". It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave. Not to mention there might not be any "array". A string (whatever the encoding / representation) is a c…

The python doc says "str" are immutable sequences of unicode code points. Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing). I couldn't find out in the documentation whether the __getitem__ is O(1), which I consider a deficiency -- this should definitely be well documented. It doesn't really matter how some people think "how people understand" something, the document…

Except a sequence is not an array.

So OP’s definition still does not apply to python’s definition of a string in an unambiguous manner, which was the claim they were making.

In fact, using the OP’s “unambiguous” definition leads to the conclusion that strings shouldn’t have a length function at all since it’s not an array.

Post reply on HN