Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

121–130 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#121

Post by temac »

Dylan16807 wrote: post #30
dahfizz wrote: post #26

Maybe not wrong, but it's the worst option. 5 is the number of code points, and 17 is the number of bytes. Both are reasonable answers. 7 is the number of code units for utf-16. Seems like the least useful option.

That means 7 is also a measure of bytes, just slightly more awkward. So it's roughly on par with 17. For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.

I think that argument makes as much sense as saying that an engine is less useful than a car. And pretending that engine.weight should return the weight of the car.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#122

Post by namaria »

I have read somewhere that you should learn 2 or 3 programming languages from the get go. If you learn one, you run the risk of letting it's shape dictate how you mentally model computation. At some point someone who learned a dynamically typed programming language first is bound to find out why data types matter.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#123

Post by nerdponx »

omoikane wrote: post #56

Ruby gives you the choice to iterate over all types, via `each_byte`, `each_char`, `each_codepoint`, or `each_grapheme_cluster`. https://ruby-doc.org/3.2.2/String.html#class-String-label-Me...

Python would benefit from this significantly IMO.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#124

Post by Firmwarrior »

namaria wrote: post #122

I have read somewhere that you should learn 2 or 3 programming languages from the get go. If you learn one, you run the risk of letting it's shape dictate how you mentally model computation. At some point someone who learned a dynamically typed programming language first is bound to find out why data types matter.

I started with JavaScript and went to c/logic gates/assembly, and it wasn't too bad

I had to learn what pointers were, but it was OK, haha.

That said, maybe people who don't have "The Knack" would be better off learning a slightly harder language first..

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#125

Post by nerdponx »

neallindsay wrote: post #75

Until reading this I had never heard of UTF-32. It doesn't seem like a good way to encode strings.

It's useful if you want array-like semantics (e.g. O(1) lookup) on Unicode text strings, because you have a fixed size for every codepoint, unliked UTF-8. Python for example uses it internally.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#126

Post by lelanthran »

BeefWellington wrote: post #33
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

5 makes perfect sense to me; the author's complaints seem kinda silly. An area this makes sense is, what do you expect to get if you do something like: emoji = " " print(emoji[:3]) Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes? Modern strings are complex objects that have evolved a bit past cha…

Well, an index into a string is not necessarily another string, nor a character.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#127

Post by chrismorgan »

xeeeeeeeeeeenu wrote: post #34

I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are…

> UTF-16 code units, on the other hand, are an implementation detail of UTF-16.

Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32.

UTF-32 and UTF-8 code units are truly implementation details of their encodings, as other encodings don’t need to know about them in any way. UTF-32’s code units are a trivial mapping between scalar values and 32-bit values (not four-byte values, given the big- and little-endian variants), but that still causes UTF-32 code units to be semantically distinct from Unicode scalar values. U+12345 is a Unicode scalar value and doesn’t have any “size”: it’s an abstract value. 0x00012345 is a UTF-32 code unit, a 32-bit value.

If you’re talking about encoding of Unicode scalar values, you talk about code units. Even when talking about UTF-32, the code unit/scalar value semantic distinction is worth maintaining.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#128

Post by Joker_vD »

Dylan16807 wrote: post #30
dahfizz wrote: post #26

Maybe not wrong, but it's the worst option. 5 is the number of code points, and 17 is the number of bytes. Both are reasonable answers. 7 is the number of code units for utf-16. Seems like the least useful option.

That means 7 is also a measure of bytes, just slightly more awkward. So it's roughly on par with 17. For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.

7 cannot be a measure of bytes because a UTF-16 point takes 2 bytes, so the number has to be even. Did you mean 14?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#129

Post by pajko »

arcticbull wrote: post #24
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> Not until they have to explain to their designer why they can't limit a label to '10 characters.'

Or in a single font. It's impossible to render any mixed combination of simplified Chinese, traditional Chinese and Japanese with a single font (Korean might be also involved, but not sure about that). Even in Unicode, characters might share the the same space which don't have anything common in their looks, nor in their meaning. That applies to the shared CJK space as well. Btw. Japanese has halfwidth and fullwidth characters.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#130

Post by farhanhubble »

secret-noun wrote: post #53

Related: I wrote a little web app that lets you see the codepoints for text like this https://unicode-x-ray.com/?t=%F0%9F%A4%A6%F0%9F%8F%BC%E2%80%... (sorry if link looks scary, that's just the URL encoding of this emoji)

I worked on Unicode parsers for several years and the single most indispensable tool was https://r12a.github.io/app-conversion/
Post reply on HN