Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

131–140 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#131

Post by Joker_vD »

mjevans wrote: post #116
masklinn wrote: post #110

Earlier quoted context omitted.

Windows, Java, C#, javascript, a surprising number of XML documents (though less so as time marches on thankfully), ICU I think uses UTF-16 internally (for the same historical reasons as the other 4), JOLIET file names are UCS2, some phones interpret “16-bit” SMS as UTF-16 (the spec says UCS2). > and BOTH of those are insane for sticking to it They don’t really have much of a choice because they exposed those semanti…

I'm not a Windows based programmer, but couldn't they leave the old API's in place, but make UTF-8 safe versions available for everyone and switch to that... E.G. with Win 11?

You can set the system codepage to CP_UTF8 since Win 10, I guess, although IIRC it still doesn't work for input. But a) there is a lot of programs using A() functions that don't expect that and break in subtle ways, e.g. DBCS-encoding-aware programs suddenly break because they don't expect a codepoint to span for more than 2 bytes; b) most of the sanely written programs either use UTF-16 explicitly, or use UTF-8 internally and convert between UTF-8 and UTF-16 before/after calling W() functions.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#132

Post by arcticbull »

rightbyte wrote: post #120
arcticbull wrote: post #24

Earlier quoted context omitted.

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> for western programmers And by western you mean american, right? You can't even use ASCII in the UK --- '£'.

Yes I was using western as synonymous with ASCII users, and it’s not. My bad.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#133

Post by mjburgess »

rightbyte wrote: post #120
arcticbull wrote: post #24

Earlier quoted context omitted.

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

> for western programmers And by western you mean american, right? You can't even use ASCII in the UK --- '£'.

£ is still just a byte

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#134

Post by alkonaut »

Really the correct way to design string APIs would be to not have an ambiguous "length" at all, but to always require specifying whether you want UTF8-bytes, memory bytes, code points, graphemes, whatever.

However such an API would be pretty cumbersome because for all non-edge cases (read: a western language and a reasonable encoding that language - which when looking at world demographics is a very narrow way of saying non-edge case) we just want to ignore all that fancy stuff and assume it's latin-1/ascii and use "Length" and get on with it, usually accepting that it doesn't work for many scripts or emoji.

So almost every api I have encountered has both the dangerous or ambiguous "length" and any number of the more specific counts. Good? No. But good enough, I guess.

A much worse related API that exists every where is that for parsing and formatting numbers to and from text. How that's done "depends" but most languages I have seen - unfortunately - offers a "default way". In the worst examples - looking at you .NET - this default uses the system env and assumes formatting and parsing numbers should us the OS locale. Horrible horrible idea when used in conjunction with automatic type conversions. WriteLine($"The size is {3.5}"); shouldn't print "3.5" in the US and "3,5" somewhere else.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#135

Post by Joker_vD »

nerdponx wrote: post #125
neallindsay wrote: post #75

Until reading this I had never heard of UTF-32. It doesn't seem like a good way to encode strings.

It's useful if you want array-like semantics (e.g. O(1) lookup) on Unicode text strings, because you have a fixed size for every codepoint, unliked UTF-8. Python for example uses it internally.

And it compresses just as well as UTF-8 for transfer/storage purposes.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#136

Post by dchest »

ceeam wrote: post #71

These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.

If not for emoji, lots of software wouldn't care about correctly processing strings in other languages, so it's good that we have them.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#137

Post by edent »

mjburgess wrote: post #133
rightbyte wrote: post #120

Earlier quoted context omitted.

> for western programmers And by western you mean american, right? You can't even use ASCII in the UK --- '£'.

£ is still just a byte

And yet it is reasonably common to see "£" when the UTF-8 is misinterpreted.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#138

Post by johannes1234321 »

mjburgess wrote: post #133
rightbyte wrote: post #120

Earlier quoted context omitted.

> for western programmers And by western you mean american, right? You can't even use ASCII in the UK --- '£'.

£ is still just a byte

When using latin-1/latin-15/iso-8859-1/iso-8859-15/cp1252 that statement is true. With utf-8 it is two bytes (c2 a3), if a software uses utf-16, ucs-2, etc. it may be more.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#139

Post by chrismorgan »

hgs3 wrote: post #15

Python 3's approach is the most correct: Unicode defines text as a sequence of code points. UTF-whatever is an implementation detail.

Python 3’s approach snatched defeat from the jaws of victory.

They aimed to work with a nice, clean, abstract concept, untrammelled by encoding squabbles. They failed badly by choosing code units rather than scalar values (Unicode strings are sequences of scalar values, not code points—'\udead' is a valid Python string, but you can’t encode it into any UTF-* format since [U+DEAD] is not a valid Unicode string).

Then they also neglected to observe that they were optimising for something that you should practically never be doing, so that now everyone has to pay the costs. As the article summarises it part-way through: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”

Seriously, Python 3’s approach is almost the worst of all available worlds. I loathe UTF-16 with such fiery passion that I can’t quite bring myself to say Python 3’s approach is worse than weak UTF-16, but it’s of similar badness in practical terms. The decisions were very clearly made by people that were not expert in the domain and who were caught up in a Concept of Mathematical Purity. They’ve since walked some of it back as far as they could, and I think did recognise it all as a mistake (no citation, just a vague memory of seeing such an admission), but they can’t fix it all properly without a breaking change.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#140

Post by L3viathan »

masklinn wrote: post #103
tehsauce wrote: post #20

Earlier quoted context omitted.

Why is it the worst way though?

Because it’s never actually useful. You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text. A length in codepoints gives you noth…

It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts.
Post reply on HN