Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emoji.length == 2

blog.jonnew.com

Post reply on HN
Print view View original

151–160 of 166 posts

Re: Emoji.length == 2

#151

Post by Dylan16807 »

burntsushi wrote: post #146
Dylan16807 wrote: post #102

Earlier quoted context omitted.

> But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points. Not at all. I've never seen people using UTF-8 deal with a code unit stage. They parse directly from bytes to code points. > While for ex…

> I thought that was an invalid code point. Surrogate codepoints are indeed valid codepoints. It's just that valid UTF-8 is not allowed to encode surrogate codepoints, so the space of codepoints supported by UTF-8 is actually a subset of all Unicode codepoints. This subset is known as the set of Unicode scalar values. ("All codepoints except for surrogates.")

Those points cannot be validly encoded in any format. I suppose you can argue that they are valid-but-unusable in an abstract sense, since the unicode standard does not actually label any code points as valid/invalid, but if you were going to label any code points as invalid then those would be in the group.

Re: Emoji.length == 2

#152

Post by lsv1 »

As a developer dealing with the encoding of user input made in UTF-8 into a legacy systems which only support ASCII... I prefer this.

Re: Emoji.length == 2

#153

Post by Sean1708 »

innocenat wrote: post #88
yongjik wrote: post #70

Earlier quoted context omitted.

I believe that was a necessary compromise to use Hangul on any software not authored by Koreans. "These are characters from a country you've never been to. Each three-byte sequence (assuming UTF-8) corresponds to a square-shaped character." --> Easy for everyone to understand, and less chance of screwup (as long as the software supports any Unicode at all). "These should be decomposed into sequences of two or three c…

My Linux Chrome shows your example perfectly though. Note that I also have CJK language pack installed.

My Linux Firefox also behaves correctly, but I don't have any language packs installed AFAIA.

Re: Emoji.length == 2

#154

Post by username223 »

darkengine wrote: post #8

The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. To use the author's example: ‍woman - 1 codepoint black woman - 2 codepoints, w…

> woman kissing woman - 7 codepoints, woman + ZWJ + heart + ZWJ + kissy lips + ZWJ + woman

LOL, I knew about the crazy flag characters, but I had no idea it was this bad. Does "woman + ZWJ + heart + ZWJ + hands pressed together + ZWJ + woman" become "2 women lovingly holding hands"? Unicode has become completely absurd, and I am grateful every day that I'm not one of the poor coders having to implement it.

Humorously enough, input methods have not advanced at all. To type any of these things, I need to open a character picker, then either type in the character's name if I know it, or scroll through pages of symbols until I find the one I want. Yet we still call this "text."

Re: Emoji.length == 2

#155

Post by Sean1708 »

paulddraper wrote: post #95
toast0 wrote: post #66

Earlier quoted context omitted.

> Perhaps someone can shed light on a compelling use case for knowing the number of grapheme clusters in a particular string, because I haven't been able to think of one. If you have a limit on the length of a field, it helps to tell the user what it is in a way they understand. For non-technical users, bytes (and the embedded issue of encoding) and code points are both pretty esoteric, but number of symbols is less…

Why do you have a limit on the length of a field? So it can fit in a database, i.e. with a certain number of bytes?

If that's why you have a limit then please go and change that immediately.

No, this post is talking about having a minimum length on the password for safety reasons (i.e. a limit on the minimum entropy). You're right that a minimum byte length will ensure this, but what happens when your user types in n-1 "things" but their password gets accepted anyway. That's only a minor thing but (and I'm not entirely sure whether this is possible) what about when your user types in n "things" but the password doesn't get accepted because it's actually only n-1 bytes. Now the password won't be accepted and the user has no idea why.

I agree that these are relatively trivial things, but the point is that it's not as simple as "just use the byte length".

Re: Emoji.length == 2

#156

Post by danbruc »

Dylan16807 wrote: post #151
burntsushi wrote: post #146

Earlier quoted context omitted.

> I thought that was an invalid code point. Surrogate codepoints are indeed valid codepoints. It's just that valid UTF-8 is not allowed to encode surrogate codepoints, so the space of codepoints supported by UTF-8 is actually a subset of all Unicode codepoints. This subset is known as the set of Unicode scalar values. ("All codepoints except for surrogates.")

Those points cannot be validly encoded in any format. I suppose you can argue that they are valid-but-unusable in an abstract sense, since the unicode standard does not actually label any code points as valid/invalid, but if you were going to label any code points as invalid then those would be in the group.

You are certainly correct that it is common to not pay too much attention to what things are called in the specification, especially if you want to create a fast implementation. Logically you will still go through all the layers even if you operate on only one physical representation.

My admittedly quite limited experience with Unicode is from trying to exploit implementation bugs. And with that focus it is quite natural to pay close attention to the different layers in the specification in order to see where optimized implementations might miss edge cases.

And I am generally a big fan of staying close to the word of standards, if it does not cause unacceptable performance issues, I would always prefer to stick with the terminology of the standard even if it means that there will be transformations that are just the identity.

The distinction between code points and scalar values will for example become relevant if you implement Unicode meta data. There you can query meta data about surrogate code points even if a parser should never produce those code points.

Re: Emoji.length == 2

#157

Post by StringEpsilon »

xnyhps wrote: post #118
tomjakubowski wrote: post #100

Earlier quoted context omitted.

> A reasonable test for passwords is to run them through an IDNA checker, which checks whether a string is acceptable as a domain name component. This catches most weird stuff, such as mixed left-to-right and right-to-left symbols, zero-width markers, homoglyphs, and emoji. Why test this at all? It's not as if a website should ever need to render a user's password as text. Is there another use case for excluding this…

Suppose I include 'ü': LATIN SMALL LETTER U WITH DIAERESIS in my password. I switch to a different browser/OS/language and now when I enter "ü" I get 'u': LATIN SMALL LETTER U + ' ̈': COMBINING DIAERESIS. I can't log in anymore, though what I do is identical and defined to be equivalent. Especially if the password is hashed before comparing it, you can't treat it as just a sequence of bytes. You don't need to use IDN…

I would not actually disallow these characters, but you may warn the user about the existance of problematic characters in their password of choice.

If I want to use äöüßÄÖÜẞ because I'm confident that I can properly type them on all devices I'll need to type then, then let me. It's not your concern what method of input I'm using.

And maybe, just maybe, using latin characters is actually more of a hassle for a user anyway. (I think the risk of that occoring is low, but still. At the moment, it's a self-fulfilling prophecy that all users have proper method to input atin script available. We simply force them to have one.)

Edit: And the confusion is also possible with just latin characters. U+0430 looks exactly like "a", but has a different code point and thus ruins the hash.

Re: Emoji.length == 2

#158

Post by pbhjpbhj »

masklinn wrote: post #132
pbhjpbhj wrote: post #80

Earlier quoted context omitted.

On Ubuntu, and probably other OS/distro too, ctrl+alt+shift+U gives you a underline-u symbol, type in the Unicode and then press .

I'm not sure what use that would be? U+ is a normal way to designate codepoints, and I can't put the actual emoji in the comment as they're stripped on submission.

Ah, sorry, that comment was for those playing along who might want to see the listed glyphs.

Re: Emoji.length == 2

#159

Post by ggchappell »

danbruc wrote: post #5

The Unicode standard describes in Annex 29 [1] how to properly split strings into grapheme clusters. And here [2] is a JavaScript implementation. This is a solved problem. [1] http://www.unicode.org/reports/tr29/ [2] https://github.com/orling/grapheme-splitter

> And here [2] is a JavaScript implementation.

Is it up to date?

The last commit to this repo was on July 16, 2015, and the code says it conforms to the 8.0 standard. But Unicode 9.0 came out in June 2016. The document in your link [1] indicates that there were changes in the text-segmentation rules in the 9.0 release. However, I can't say whether any of these affect the correctness of the code.

Re: Emoji.length == 2

#160

Post by pc2g4d »

Just ran into this yesterday when I discovered that an emoji character wouldn't fit into Rust's `char` type. I just changed the type to `&'static str` but I still wish there was a single `grapheme` type or something like that.
Post reply on HN