Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

201–210 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#201

Post by jodrellblank »

2h wrote: post #25

this one one of those things that people point to when comparing languages, but in reality rarely matters. with Go, you just get the number of bytes, which the the correct default thing to do: https://godocs.io/builtin#len if the language default was anything other than this, THAT WOULD BE WRONG and unexpected. I would prefer the default to be the dumb, fast thing. then if I want the slow, fancy thing, I can import s…

For anyone wondering what Go does, it looks like Python2's way[1]; strings are byte sequences with no guarantees of UTF{anything} correctness. Go's source code is specified to be UTF8 so string literals in source code will become valid UTF8 encoded strings, but any string from any library call or code you didn't write might contain invalid Unicode text, or mixed encodings, or anything.

That feels a bit "pit of despair" design[2], the default thing is unhelpful and doing more than that requires the programmer to climb up out of it.

[1] https://go.dev/blog/strings

[2] https://blog.codinghorror.com/falling-into-the-pit-of-succes...

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#202

Post by BeefWellington »

coldtea wrote: post #118
BeefWellington wrote: post #33

Earlier quoted context omitted.

5 makes perfect sense to me; the author's complaints seem kinda silly. An area this makes sense is, what do you expect to get if you do something like: emoji = " " print(emoji[:3]) Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes? Modern strings are complex objects that have evolved a bit past cha…

> Should this throw an error because there's only one displayed "character"? Absolutely.

I think this is where the misunderstanding comes in. Python doesn't treat strings as char[] but as essentially unicode_codepoint[].

Whether this is a good idea on the whole is debatable, there's even a full PEP talking about the security concerns around doing it this way[1].

However, given this is how it works, the behaviour displayed makes complete sense to me and is the best of the bad choices presented by needing multi-byte strings.

[1]: https://peps.python.org/pep-0672/

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#203

Post by BeefWellington »

masklinn wrote: post #113
BeefWellington wrote: post #33

Earlier quoted context omitted.

5 makes perfect sense to me; the author's complaints seem kinda silly. An area this makes sense is, what do you expect to get if you do something like: emoji = " " print(emoji[:3]) Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes? Modern strings are complex objects that have evolved a bit past cha…

> Should this throw an error because there's only one displayed "character"? Why should it not? You’re literally breaking the content. Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. > Modern strings are complex objects that have evolved a bit past char[] or byte[]. And yet that’s exactly what you’re advocating, just with 21 bit chars.

> Why should it not? You’re literally breaking the content.

Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints.

> Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue.

Substring is a broken operation? What's the justification for that idea?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#204

Post by amake »

tremon wrote: post #162
amake wrote: post #152

Earlier quoted context omitted.

It's ambiguous because it's not clear what elements go into separate cells of the array.

Why is it ambiguous? The Python documentation is pretty clear about what type of elements a string contains: > Strings are immutable sequences of Unicode code points (from https://docs.python.org/3/library/stdtypes.html#text-sequenc... )

Did you not read the article?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#205

Post by kps »

cylemons wrote: post #191
chrismorgan wrote: post #127

Earlier quoted context omitted.

> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates , and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as ot…

I don't see what's inherently wrong with UTF-16 surrogates. If I am not wrong, a given UTF-16 codeunit is unambigously either a complete code point, a first surrogate, or a second surrogate. Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? I don't see anyone trying to represent invalid utf-8 in utf-16 or 32.

> Why should we expect invalid utf-16 strings to be representable in utf-8 or 32?

We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF are used in UTF-8 encoding.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#206

Post by boxed »

masklinn wrote: post #103
tehsauce wrote: post #20

Earlier quoted context omitted.

Why is it the worst way though?

Because it’s never actually useful. You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text. A length in codepoints gives you noth…

But 100% of all those complaints also apply to JS, except that UTF-16 is just much more stupid than UTF-32?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#207

Post by adgjlsfhk1 »

Spivak wrote: post #183
adgjlsfhk1 wrote: post #17

Earlier quoted context omitted.

Unicode defines text as a number of different types of things. They are sequences of codepoints, sequences of graphemes, sequences of graphime clusters. Furthermore, codepoints are different depending on how you normalize them. Accented characters can be written two different ways and have a different number of codepoints depending on how you write them (and if normalization is used)

Grapheme are a made up human thing that, while useful, is locale dependent. Most people when they talk about grapheme clusters mean the default "locale-independent" graphemes but it's not the only one (in Hungarian for example 'ly' is a single letter). Having the same string be two different lengths in two countries is… let's go with surprising. The common denominator where everyone computes the same number is code p…

Except they won't (if they are doing normalization). à and à have different numbers of codepoints (the first is 0x00E0, the second is 0xc3 0xa0).

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#208

Post by zkldi »

PhilipRoman wrote: post #186

I cannot think of a single common case where grapheme cluster count is important. If you want to print them aligned to a terminal - guess what, double width characters exist, so the only reliable way is to print them first, measure the cursor movement using escape sequences, calculate length and erase the originally printed data. Even for limiting input field sizes byte count is much better, as otherwise you are open…

this doesn't "just work" at all?

utf-8 is a variable width encoding and if you treat it like ascii your software just isn't going to work outside of the ascii range.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#210

Post by kps »

Findecanor wrote: post #165
ceeam wrote: post #71

These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.

I agree that they introduced unnecessary complexity for text encoding, and for font-rendering (which are expected to support multi-coloured emoticons now). I once started writing on a text editor, and then fell deep into Unicode handling. I have now spent more work on the Unicode parts than on anything else in the program. I think that the industry could have instead adopted the old web-forum convention of colon-word…

Naturally Unicode did the equivalent of colon-encoding too, for flags.
Post reply on HN