Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

261–270 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#261

Post by masklinn »

BeefWellington wrote: post #203
masklinn wrote: post #113

Earlier quoted context omitted.

> Should this throw an error because there's only one displayed "character"? Why should it not? You’re literally breaking the content. Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. > Modern strings are complex objects that have evolved a bit past char[] or byte[]. And yet that’s exactly what you’re advocating, just with 21 bit chars.

> Why should it not? You’re literally breaking the content. Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints. > Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. Substring is a broken operation? What's the justification for that idea?

> Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints.

"Ice trays are just a pile of molecules rather than "cubes", so all I'm doing is separating those molecules", he states as he activates the igniter.

> Substring is a broken operation? What's the justification for that idea?

You take a thing and you mangle beyond recognition without regards for its purpose or meaning. That's like considering the jaws of life a normal part of opening a door to take a piss at work.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#262

Post by JZerf »

ElectricalUnion wrote: post #253
rzwitserloot wrote: post #250

Java loaded full unicode code point semantics into its standard `java.lang.String` class. These _are not guaranteed_ to have `O(1)` performance characteristics, because the underlying storage format is dynamically either a UTF-16-esque variant (with surrogate pairs for characters that don't fit in 16 bit), or a single-byte-per-char format if the string does not contain any non-ASCII. This has the advantage of being v…

> The provided java snippet is vanilla valid Did not you still need the `java --source 11 ${filename_without_java_extension_because_JEP_330}` to use it? And you still need a wrapper class with a static method main in it.

I was a little puzzled by this compared to what I was used to with Java in the past. It looks like the grandparent's code relies on JEP 445 ( https://openjdk.org/jeps/445 ) which is a preview feature as was mentioned but it also apparently requires the very latest Java 21 which hasn't even been officially released yet.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#263

Post by Bjartr »

ElectricalUnion wrote: post #253
rzwitserloot wrote: post #250

Java loaded full unicode code point semantics into its standard `java.lang.String` class. These _are not guaranteed_ to have `O(1)` performance characteristics, because the underlying storage format is dynamically either a UTF-16-esque variant (with surrogate pairs for characters that don't fit in 16 bit), or a single-byte-per-char format if the string does not contain any non-ASCII. This has the advantage of being v…

> The provided java snippet is vanilla valid Did not you still need the `java --source 11 ${filename_without_java_extension_because_JEP_330}` to use it? And you still need a wrapper class with a static method main in it.

> And you still need a wrapper class with a static method main in it

One of the preview features he's using is JEP 445[1] that allows you to omit the wrapper class, as well as the arguments to main and the public and static modifiers.

[1]https://openjdk.org/jeps/445

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#264

Post by kps »

cylemons wrote: post #258
kps wrote: post #205

Earlier quoted context omitted.

> Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF ar…

Is having a hole from U+D800 to U+DFFF such a big deal? The parent comment was specifically talking about surrogate pairs. That to me looks more like buggy implementation issue rather than standards issue.

The main issue is that it adds validation code (if one is sticking to the standard) for things that don't care about UTF-16 at all.

It does occupy 1/32 of the BMP, displaying a couple thousand potential actual characters (making them take an extra byte in UTF-8, and an extra two in UTF-16).

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#265

Post by pierrebai »

chrismorgan wrote: post #147
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

> For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.” “Not needing to know the byte size semantics” seems reasonable, but it simply isn’t a useful goal. The things it makes easier or faster (knowing how many c…

... but maybe it simplifies and speed-up the internal processing? I haven't looked at Python 3 C implementation of strings, but that is a guess. Also, IIRC, Python 3 has the ability to keep different internal representations of strings and uses the most compact one. If all character are 7-bit ASCII, it uses bytes representation. That's what I remember from Python dev discussions long ago.

But the overall tone of the article is still bashful. Caring about the internal representation of strings and bashing UTF-32 feels lame and angry. (Especially if I'm right about the multi-rep nature of Python 3: their choice is good for most text, and they could add a UTF-8 internal rep in the future, although that would probably break enough code that expect the UTF-32 value for len() that it is not worth it.)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#266

Post by chrismorgan »

cylemons wrote: post #258
kps wrote: post #205

Earlier quoted context omitted.

> Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF ar…

Is having a hole from U+D800 to U+DFFF such a big deal? The parent comment was specifically talking about surrogate pairs. That to me looks more like buggy implementation issue rather than standards issue.

As a hole, it would only be annoying and a performance penalty for validation. But by its very design, it will leak, and it does in such ways that it became the worst thing to ever happen to Unicode. I don’t know of a single language or library that uses UTF-16 for strings that validates strings: every last one actually uses sequences of UTF-16 code units, potentially ill-formed, and has APIs that guarantee this will leak to other systems. This has caused a lot of trouble for environments that then try to work with the vastly more sensible UTF-8 (the only credible alternative for interchange). Servo, for example, wanted to work in UTF-8, for massive memory savings and performance improvements, but the web has built on and depends on UTF-16 code unit semantics so much that they had to invent WTF-8, which is basically “UTF-8 but with that hole filled in” (well, actually it’s more complicated: half filled in, permitting only unpaired surrogates, so that you still have only one representation).

So: the problem is that the Unicode standard was compromised for the sake of a buggy encoding (they should instead have written UCS-2 off as a failed experiment), and every implementation that uses that buggy encoding is itself buggy, and that bugginess has made it into many other standards (e.g. ECMAScript).

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#267

Post by alkonaut »

hardware2win wrote: post #197
alkonaut wrote: post #134

Really the correct way to design string APIs would be to not have an ambiguous "length" at all, but to always require specifying whether you want UTF8-bytes, memory bytes, code points, graphemes, whatever. However such an API would be pretty cumbersome because for all non-edge cases (read: a western language and a reasonable encoding that language - which when looking at world demographics is a very narrow way of say…

>Horrible horrible idea when used in conjunction with automatic type conversions. WriteLine($"The size is {3.5}"); shouldn't print "3.5" in the US and "3,5" somewhere else. Actually why?

Because it’s only (maybe) a good design if it’s to be read by a human but that’s not a very general case. Instead people unknowingly make for example some exporter for a text format and write code that writes "X={x_coord}" and it passes all the unit tests and all the acceptance tests and then it breaks once it hits a French or scandinavian machine.

A great example how bad it is would be that the C# compiler repo for a very long time had tests that failed for everyone with non-US formatting.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#268

Post by hnfong »

mjevans wrote: post #100
Dylan16807 wrote: post #73

Earlier quoted context omitted.

Lots of systems use UTF-16 internally and externally. Counting bytes in UTF-16 is, on average, almost as useful as counting bytes in UTF-8. I don't think just about anything communicates in UTF-32. 5 is basically just a codepoint count, and as such I don't think its usefulness rating should be between the byte counts.

Only Windows and Java come to mind - and BOTH of those are insane for sticking to it when the entire rest of the world has moved on.

I hate UTF-16 and the systems that use it with a passion, but...

Windows and Java (and Javascript) adopted unicode at a time when it was thought that 64k code points would be enough for everyone. Then they prioritized backwards compatibility over anything else. Most of us have benefited from their insistence on backwards compatibility in some form or the other, so I'm really not in a position to complain about it :-/

That said, IMHO any "length" property (as opposed to `codepoints` or `bytes`) on a UTF-16 string should definitely be deprecated.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#269

Post by hnfong »

dchest wrote: post #136
ceeam wrote: post #71

These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.

If not for emoji, lots of software wouldn't care about correctly processing strings in other languages, so it's good that we have them.

This.

I've been quite happy that popular emojis were introduced in supplementary planes, because my language has quite a few common words (eg. 𨋢 [lift/escalator]) that ended up on plane 2.

Proper software support for those characters used to be terrible, but things got much better after emojis became popular. So, thanks and sorry everyone :)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#270

Post by hnfong »

kps wrote: post #213
ceeam wrote: post #71

These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.

Unification was reasonable at the time, given the goal to fit Unicode in 16 bits, and willingness to exclude obsolete characters. It's just that they followed official Japanese standards, and therefore unified too many from the point of view of other languages. I think the first big mistake was using postfix/infix operators (combining characters, modifiers, variant selectors, joiners, etc.) rather than prefix, prefer…

> It's just that they followed official Japanese standards, and therefore unified too many from the point of view of other languages.

And they're still complaining about the handful of cases that were missed: https://news.ycombinator.com/item?id=29022906

----

Another mistake IMHO was that they accepted too many "dictionary characters", i.e. the ones only seen once or twice in some obscure dictionary -- they often had explanations like "an obscure form of [common character]".

Post reply on HN