Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emoji.length == 2

blog.jonnew.com

Post reply on HN
Print view View original

101–110 of 166 posts

Re: Emoji.length == 2

#101

Post by danbruc »

Dylan16807 wrote: post #90
deathanatos wrote: post #78

Earlier quoted context omitted.

A "code unit" exists in UTF-8 and UTF-32; they are not unique to UTF-16.[1] UTF-8's relationship with code points is approximately the same as UTF-16's, except that UTF-8 systems tend to understand code points better because if they didn't, things break a lot sooner, whereas they mostly work in UTF-16. Your entire argument that graphemes are a poor way to deal with unicode seems to be that current programming languag…

> A "code unit" exists in UTF-8 and UTF-32; they are not unique to UTF-16. Technically yes. But they are only "exposed" in UTF-16. In UTF-32 code points and code units are the same size, so you only have to deal with code points. In UTF-8 you only have to deal with code points and bytes. UTF-16 is unique in having something that is neither code point nor byte but sits in between.

That is certainly true if you only look at the word sizes at different layers. But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points.

While different layers may use words of the same size, there are still differences, for example what is valid and what is not. While for example U+00D800 is a perfectly fine code point, the first high-surrogate, 0x0000D800 is not a valid UTF-32 code unit. 0xC0 0xA0 is a perfectly fine pair of bytes, both are valid UTF-8 code units, and they could become the code point U+000020 if only 0xC0 0xA0 were not an invalid code unit subsequence.

So yes, while I agree that UTF-16 is special in that sense that one has to deal with 8, 16 and 32 bit words, I don't think that one should dismiss the concept of code units for all encoding forms but UTF-16. There enough subtle details between the different layers so that the distinction is warranted. And that is actually something I really like about the Unicode standard, it is really precise and doesn't mix up things that are superficially the same.

Re: Emoji.length == 2

#102

Post by Dylan16807 »

danbruc wrote: post #101
Dylan16807 wrote: post #90

Earlier quoted context omitted.

> A "code unit" exists in UTF-8 and UTF-32; they are not unique to UTF-16. Technically yes. But they are only "exposed" in UTF-16. In UTF-32 code points and code units are the same size, so you only have to deal with code points. In UTF-8 you only have to deal with code points and bytes. UTF-16 is unique in having something that is neither code point nor byte but sits in between.

That is certainly true if you only look at the word sizes at different layers. But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points. While different layers may use words of the same size, there…

> But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points.

Not at all. I've never seen people using UTF-8 deal with a code unit stage. They parse directly from bytes to code points.

> While for example U+00D800 is a perfectly fine code point, the first high-surrogate, 0x0000D800 is not a valid UTF-32 code unit.

I thought that was an invalid code point. Where would I look to see the difference? Nevertheless I would expect most code to make no distinction between the invalidity of 0x0000D800 and 0x44444444, except perhaps to give a better error message.

> 0xC0 0xA0 is a perfectly fine pair of bytes, both are valid UTF-8 code units, and they could become the code point U+000020 if only 0xC0 0xA0 were not an invalid code unit subsequence.

If you say that they're correct code units then at what point do you distinguish bytes and code units? In practice almost nobody decodes UTF-8 with an understanding of code units, neither by that name nor any other name. They simply see bytes that correctly encode code points, and bytes that don't.

Especially if you say that C0 is a valid code unit despite it not appearing in any valid UTF-8 sequences.

Re: Emoji.length == 2

#103

Post by throwaway399 »

deathanatos wrote: post #78
Joeri wrote: post #37

Earlier quoted context omitted.

This is most definitely not a solved problem, because graphemes (visual symbols) are a poor way to deal with unicode in the real world. Pretty much all systems either deal with the length in bytes (if they're old-style C), in code units / byte pairs (if they're UTF-16 based, like windows, java and javascript), or in unicode code points (if they're UTF-8 based, like every proper system should be). Dealing with the len…

A "code unit" exists in UTF-8 and UTF-32; they are not unique to UTF-16.[1] UTF-8's relationship with code points is approximately the same as UTF-16's, except that UTF-8 systems tend to understand code points better because if they didn't, things break a lot sooner, whereas they mostly work in UTF-16. Your entire argument that graphemes are a poor way to deal with unicode seems to be that current programming languag…

How would you implement graphemes compatibility if you can have unlimited number of code points combine into a grapheme? Designing an efficient storage solution for such text seems like a nightmare.

Any time you define an upper limit, someone will come up with more emojis that will require larger number of code points per grapheme.

Re: Emoji.length == 2

#104

Post by lifthrasiir »

duskwuff wrote: post #13
melloclello wrote: post #9

Earlier quoted context omitted.

Man, imagine if you could compose chinese characters out of radicals like this. I'm not sure if that would be a good thing or a bad thing.

The one that still surprises me is Hangul (Korean script). Hangul characters are made of 24 basic characters (jamo) which represent consonant and vowel sounds, which are composed into Hangul characters representing syllables. Unicode has a block for Hangul jamo, but they aren't used in typical text. Instead, Hangul are presented using a massive 11K-codepoint block of every possible precomposed syllable. ¯\_(ツ)_/¯

This is primarily because the legacy character set---KS X 1001---already contained tons (2,350 to be exact) of precomposed syllables. Unicode 1.0 and 1.1 had lots of syllables encoded in this way, with no good way to figure out the pattern, and in 2.0 the entire Hangul syllable block is reallocated to a single block of 11,172 correctly [1] ordered syllables.

So yeah, Unicode is not a problem here (the compatibility with existing character sets was essential for Unicode's success), it's a problem of legacy character sets :-)

[1] Only correct for South Koreans though :) but the pattern is now very regular and it's much more efficient than heavy table lookups.

Re: Emoji.length == 2

#105

Post by lifthrasiir »

Manishearth wrote: post #34
Animats wrote: post #24

Before emoji, fonts and colors were independent. Combining the two creates a mess. Try using emoji in an editor with syntax coloring. We got into this because some people thought that single-color emoji were racist.[1] So now there are five skin tone options. The no-option case is usually rendered as bright yellow, which comes from the old AOL client. They got it from the happy-face icon of the 1970s. Here's the curr…

Unicode does not require fonts to use color, and the spec does try to deal with the case where you don't want to use color; explicitly talking about black-and-white renderings in multiple places. This is no different for the skin tone modifiers, it's perfectly okay to fall back to a greyscale emoji (indeed, it might make sense to render all emoji in greyscale or B&W in a text editor).

> [...] it's perfectly okay to fall back to a greyscale emoji [...]

...if you want to destroy the user experience. Practically many implementations would be forced to support colors. (I'm personally okay with that, but just to be correct)

Re: Emoji.length == 2

#106

Post by seanp2k2 »

darkengine wrote: post #8

The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. To use the author's example: ‍woman - 1 codepoint black woman - 2 codepoints, w…

>"edit: looks like HN strips emoji? Changed the emoji in the example into English words. They are all supposed to render as a single "character"."

Ironic huh, that we're discussing this on a site which has "solved" this by just pretending it doesn't exist :)

We're still on that IPv4 unicode, that Python 2.7 world, where we use scanf() to read user input and goto wherever we please.

Re: Emoji.length == 2

#107

Post by nercht12 »

jcranmer wrote: post #55
marcosdumay wrote: post #51

Earlier quoted context omitted.

> "UTF-32" which has the advantage of being fixed width It's fixed width for now. It can not hold all the current available code-points, so it will probably have the same fate as UTF-16 (but it will probably take a long time). I'd stay away from it.

There are currently 17 × 65536 code points (U+0000..U+10FFFF) in Unicode. UTF-32 could theoretically encode up to a hypothetical U+FFFFFFFF and still be fixed-width. Note that, at present, only 4 of the 17 planes have defined characters (Planes 0, 1, 2, and 14), two are reserved for private use (15 and 16), and an additional is unused but is thought to be needed (Plane 3, the TIP for historic Chinese script predecess…

The bummer is when you want to create a font that supports all the characters. Ugh. Talk about alot of work.

Re: Emoji.length == 2

#109

Post by Mithorium »

tomjakubowski wrote: post #100
Animats wrote: post #24

Before emoji, fonts and colors were independent. Combining the two creates a mess. Try using emoji in an editor with syntax coloring. We got into this because some people thought that single-color emoji were racist.[1] So now there are five skin tone options. The no-option case is usually rendered as bright yellow, which comes from the old AOL client. They got it from the happy-face icon of the 1970s. Here's the curr…

> A reasonable test for passwords is to run them through an IDNA checker, which checks whether a string is acceptable as a domain name component. This catches most weird stuff, such as mixed left-to-right and right-to-left symbols, zero-width markers, homoglyphs, and emoji. Why test this at all? It's not as if a website should ever need to render a user's password as text. Is there another use case for excluding this…

I agree, if it can live in a byte buffer and not crash the transport mechanism or hash function then it's good enough for me

Re: Emoji.length == 2

#110

Post by desdiv »

paulddraper wrote: post #95
toast0 wrote: post #66

Earlier quoted context omitted.

> Perhaps someone can shed light on a compelling use case for knowing the number of grapheme clusters in a particular string, because I haven't been able to think of one. If you have a limit on the length of a field, it helps to tell the user what it is in a way they understand. For non-technical users, bytes (and the embedded issue of encoding) and code points are both pretty esoteric, but number of symbols is less…

Why do you have a limit on the length of a field? So it can fit in a database, i.e. with a certain number of bytes?

Without a limit on password length, an attacker can DOS you by forcing you to run your KDF on gigabyte-sized strings.
Post reply on HN