Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emoji.length == 2

blog.jonnew.com

Post reply on HN
Print view View original

81–90 of 166 posts

Re: Emoji.length == 2

#81

Post by danbruc »

deathanatos wrote: post #76
danbruc wrote: post #5

The Unicode standard describes in Annex 29 [1] how to properly split strings into grapheme clusters. And here [2] is a JavaScript implementation. This is a solved problem. [1] http://www.unicode.org/reports/tr29/ [2] https://github.com/orling/grapheme-splitter

> This is a solved problem. Not … really. Yes, we "know" the solution, but the terrible APIs that compose so many language's standard string type goads the programmer into choosing the wrong method or type. JavaScript has — to an extent — the excuse of age. But the language still really (to my knowledge) lacks an effective way to deal with text that doesn't involve dragging in third-party libraries. You are not a hig…

That is what I meant, there is an existing algorithm to do this because the author tried to come up with one. That JavaScript fails to provide an implementation, well, too bad, but this is of course a problem one may have to solve in any language.

And while other languages provide the necessary support at the language or standard library level, I would guess there are quite a few developers out there that are not even aware that they are looking for enumerating grapheme clusters. But now some more know and if they made a good language choice, it is now a solved problem for them.

Re: Emoji.length == 2

#82

Post by fryguy »

TorKlingberg wrote: post #65

If you want to do Unicode correctly, you shouldn't ask for the "length" of a string. The is no true definition of length. If want to know how many bytes it uses in storage, ask for that. If you want to know how wide it will be on the screen, ask for that. Do not iterate over strings character by character.

How many dots/stars should one display for a password? That's a question that can't be answered by your two valid question. Are you suggesting that dots/stars shouldn't be displayed for passwords, since you can't ask how many "characters" it is?

Re: Emoji.length == 2

#83

Post by carapace »

phkahler wrote: post #2

Unicode is fucked. All these bullshit emojis remind me of the 1980s when ASCII was 7 bits but every computer manufacturer (Atari, Commodore, Apple, IBM, TI, etc...) made their own set of characters for the 128 values of a byte beyond ASCII. Of course Unicode is a global standard so your pile-of-poop emoji will still be a pile-of-poop on every device even if the amount of steam is different for some people. It's beyon…

Unicode is a conflation of two ideas, one good and the other impossible.

The good idea is to have a standard mapping from numbers to little pictures (glyphs, symbols, kanji, ideograms, cuneiform pokings in dried clay, scratches on a rock, whatever.) This is really all ASCII was.

The impossible idea is to encode human languages into bits. This can't be done and will only continue to cause heartache in those who try.

ASCII had English letters but wasn't an encoding for English, although you can and everyone did and does use it for that.

Re: Emoji.length == 2

#84

Post by [deleted] »

deathanatos wrote: post #78
Joeri wrote: post #37

Earlier quoted context omitted.

This is most definitely not a solved problem, because graphemes (visual symbols) are a poor way to deal with unicode in the real world. Pretty much all systems either deal with the length in bytes (if they're old-style C), in code units / byte pairs (if they're UTF-16 based, like windows, java and javascript), or in unicode code points (if they're UTF-8 based, like every proper system should be). Dealing with the len…

A "code unit" exists in UTF-8 and UTF-32; they are not unique to UTF-16.[1] UTF-8's relationship with code points is approximately the same as UTF-16's, except that UTF-8 systems tend to understand code points better because if they didn't, things break a lot sooner, whereas they mostly work in UTF-16. Your entire argument that graphemes are a poor way to deal with unicode seems to be that current programming languag…

[deleted]

Re: Emoji.length == 2

#85

Post by rspeer »

Manishearth wrote: post #43
darkengine wrote: post #11

Earlier quoted context omitted.

Somebody involved with Unicode must have had the same idea, because the ideographic description characters exist. However, I've never seen them used in practice because they don't actually render the character. You just get something like ⿰扌足, which corresponds to 捉. https://en.wikipedia.org/wiki/Ideographic_Description_Charac...

They're not supposed to render, it's purely for describing text. As are the interlinear ruby annotations.

I'm curious. Are interlinear ruby annotation codepoints actually used for their intended purpose anywhere? And what are you supposed to do when you encounter one?

I know that they appear on Unicode's shitlist in [UTR#20], a proposed tech report that contained a table of codepoints that should not be used in text meant for public consumption. UTR#20 suggested things you could do when you encounter these codepoints, but it was withdrawn, leaving the status of these codepoints rather confused.

[UTR#20]: http://www.unicode.org/reports/tr20/tr20-9.html#Interlinear

Re: Emoji.length == 2

#86

Post by rspeer »

chungy wrote: post #60

> The current largest codepoint? Why that would be a cheese wedge at U+1F9C0. How did we ever communicate before this? Sounds cute, but inaccurate. If we count the last two planes that are reserved for private use (aka, applications/users can use them for whatever domain problems they like), that would be U+10FFFD. If we count the variation selector codepoints (used for things like changing skin tone, or the look of…

I tried to look up what U+2FA1D, the highest-numbered printable character, means in context.

It is a Traditional Chinese character. It's a variant of U+2F600, 𪘀, which is pronounced "pián". It apparently is used in zero words. It's in Unicode because it's listed in the 7th section of TCA-CNS 11643-1992, a Taiwanese computing standard.

Searching for it gives lots of sites that acknowledge that it's a character that exists and then provide no definition for it.

My guess: it occurred in someone's name at some point. Pretty strange that it ended up requiring a compatibility mapping, though, when nobody seems to use the character or the character it's mapped to!

Re: Emoji.length == 2

#87

Post by [deleted] »

chungy wrote: post #60

> The current largest codepoint? Why that would be a cheese wedge at U+1F9C0. How did we ever communicate before this? Sounds cute, but inaccurate. If we count the last two planes that are reserved for private use (aka, applications/users can use them for whatever domain problems they like), that would be U+10FFFD. If we count the variation selector codepoints (used for things like changing skin tone, or the look of…

[deleted]

Re: Emoji.length == 2

#88

Post by innocenat »

yongjik wrote: post #70
duskwuff wrote: post #13

Earlier quoted context omitted.

The one that still surprises me is Hangul (Korean script). Hangul characters are made of 24 basic characters (jamo) which represent consonant and vowel sounds, which are composed into Hangul characters representing syllables. Unicode has a block for Hangul jamo, but they aren't used in typical text. Instead, Hangul are presented using a massive 11K-codepoint block of every possible precomposed syllable. ¯\_(ツ)_/¯

I believe that was a necessary compromise to use Hangul on any software not authored by Koreans. "These are characters from a country you've never been to. Each three-byte sequence (assuming UTF-8) corresponds to a square-shaped character." --> Easy for everyone to understand, and less chance of screwup (as long as the software supports any Unicode at all). "These should be decomposed into sequences of two or three c…

My Linux Chrome shows your example perfectly though. Note that I also have CJK language pack installed.

Re: Emoji.length == 2

#89

Post by kccqzy »

kmill wrote: post #35

There are multiple ways of counting "length" of a string. Number of UTF-8 bytes, number of UTF-16 code units, number of codepoints, number of grapheme clusters. These are all distinct yet valid concepts of "length." For the purpose of allocating buffers, I can see the obvious use in knowing number of bytes, UTF-16 code units, or the number of codepoints. I also see the use in being able to iterate through grapheme cl…

That forgotten use might be a special string sorting algorithm such as LSD. Or it could be a trie but input strings have many common prefixes.

Re: Emoji.length == 2

#90

Post by Dylan16807 »

deathanatos wrote: post #78
Joeri wrote: post #37

Earlier quoted context omitted.

This is most definitely not a solved problem, because graphemes (visual symbols) are a poor way to deal with unicode in the real world. Pretty much all systems either deal with the length in bytes (if they're old-style C), in code units / byte pairs (if they're UTF-16 based, like windows, java and javascript), or in unicode code points (if they're UTF-8 based, like every proper system should be). Dealing with the len…

A "code unit" exists in UTF-8 and UTF-32; they are not unique to UTF-16.[1] UTF-8's relationship with code points is approximately the same as UTF-16's, except that UTF-8 systems tend to understand code points better because if they didn't, things break a lot sooner, whereas they mostly work in UTF-16. Your entire argument that graphemes are a poor way to deal with unicode seems to be that current programming languag…

> A "code unit" exists in UTF-8 and UTF-32; they are not unique to UTF-16.

Technically yes. But they are only "exposed" in UTF-16. In UTF-32 code points and code units are the same size, so you only have to deal with code points. In UTF-8 you only have to deal with code points and bytes. UTF-16 is unique in having something that is neither code point nor byte but sits in between.

Post reply on HN