Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emoji.length == 2

blog.jonnew.com

Post reply on HN
Print view View original

131–140 of 166 posts

Re: Emoji.length == 2

#131

Post by joeblau »

I ran into this 2 years ago on Swift when I was creating an emojified version of Twitter. I wanted to ensure that each message sent had at least 1 emoji and I quickly realized that validating a string with 1 emoji was not as simple as:

  if (lastString.characters.count == 2) {
     // pseudo code to allow string and activate send button
  }
This was the app I was working on [1]; code is finished, but I'm not launching it (probably ever). The whole emoji length piece was quite frustrating because my assumption of character counting went right out of the window when I had people testing the app in Test Flight.

[1] - https://joeblau.com/emo/

Re: Emoji.length == 2

#132

Post by masklinn »

pbhjpbhj wrote: post #80
masklinn wrote: post #33

Earlier quoted context omitted.

The Love Hotel (U+1F3E9) is rather obvious, maybe the kiss mark (U+1F48B) as well, though the raunchiest ones (in actual use) are a bit more… discreet?: the aubergine (U+1F346) and splashing "sweat" (U+1F4A6).

On Ubuntu, and probably other OS/distro too, ctrl+alt+shift+U gives you a underline-u symbol, type in the Unicode and then press .

I'm not sure what use that would be? U+ is a normal way to designate codepoints, and I can't put the actual emoji in the comment as they're stripped on submission.

Re: Emoji.length == 2

#133

Post by [deleted] »

nathancahill wrote: post #96
paulddraper wrote: post #94

Earlier quoted context omitted.

> You are not a high-level language if your standard library struggles with Unicode So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages. HN teaches me something new every day.

Python 3 does pretty good.

[deleted]

Re: Emoji.length == 2

#134

Post by Manishearth »

rspeer wrote: post #124
Manishearth wrote: post #92

Earlier quoted context omitted.

> Are interlinear ruby annotation codepoints actually used for their intended purpose anywhere? Yes > And what are you supposed to do when you encounter one? Nothing. Don't display them, or display some symbolic representation. You probably shouldn't make ruby happen here; if your text is intended to be rendered correctly use a markup language. ---------- Unicode is ultimately a system for describing text. Not all st…

Can you give an example of text in the wild that uses interlinear ruby annotation codepoints? Because I searched the Common Crawl for them, and every occurrence of U+FFF9 through U+FFFB seems to have been an accident that has nothing to do with Japanese. Note that I didn't actually ask you about rendering. I care from the point of view of the base level of natural language processing. Some decisions that have nothing…

> Can you give an example of text in the wild that uses interlinear ruby annotation codepoints?

Sadly, no :( You may have luck scraping Wikibooks or some other source of PDFs or plaintext. In general you won't find interlinear annotations on the web because HTML has a better way of dealing with ruby. This is also why they're in the "shitlist", that shitlist is for stuff that's expressly not supposed to be used in markup languages.

Another way to get a good answer here is by asking the unicode mailing list, they tend to be helpful here. I know that they're used because I've heard that they are, so no first-hand experience with them. This isn't a very satisfying answer, I know, but I can't give a better one.

> Do they count as graphemes?

The annotation characters themselves? By UAX 29 they probably do, since UAX 29 doesn't try to handle many of these corner-case things (it explicitly asks you to tailor the algorithm if you care about specifics like these). ICU might deal with them better. The same goes for word segmentation, e.g. UAX 29 will not correctly word-segment Thai text, but ICU will if you ask it to. I haven't tried any of this, but it should be easy enough.

I guess a lot of this depends on what kind of processing you're doing. Ignoring the annotation sounds like the way to go for NLP, since it's ultimately an _annotation_ (which is kinda a parallel channel of info that's not essential to the text). This certainly applies for when the annotations are used for ruby, though they can be used for other things too. Interlinear annotations were almost used for the Vedic samasvara letter combiners, though they ultimately went with creating new combiners since it was a very restricted set of annotations.

They're not used much so the best way forward is probably to ignore them, really. These are a rather niche thing that never really took off.

Re: Emoji.length == 2

#135

Post by Manishearth »

joeblau wrote: post #131

I ran into this 2 years ago on Swift when I was creating an emojified version of Twitter. I wanted to ensure that each message sent had at least 1 emoji and I quickly realized that validating a string with 1 emoji was not as simple as: if (lastString.characters.count == 2) { // pseudo code to allow string and activate send button } This was the app I was working on [1]; code is finished, but I'm not launching it (pro…

Actually, this is just due to Swift not implementing Unicode 9's version of UAX 29 (which had just come out at the time). Swift should handle it correctly, but it's lagging behind in unicode 9 support. In general a "character" in a string is a grapheme cluster, and most visually-single emoji are single grapheme clusters. The exception is stuff like ‍️[1]. That should render as a male judge (I don't think there's font support for it yet) according to the spec, and it should be a single grapheme cluster, but the spec has what I consider a mistake in it where it isn't considered to be one. I've filed a bug about this, since the emoji-zwj-sequences file lists it as a valid zwj sequence, but applying the spec to the sequence gives two grapheme clusters.

There's active work now for Unicode 9 support in Swift. Since string handling is heavily dependent on this algorithm (they have a unicode trie and all for optimization!) it's trickier than just rewriting the algorithm.

But, in general, you should be able to trust Swift to do the right thing here, barring bugs like "not up to date with the spec". Swift is great like that.

[1]: https://r12a.github.io/uniview/?charlist=%F0%9F%91%A8%F0%9F%...

Re: Emoji.length == 2

#136

Post by geocar »

desdiv wrote: post #110
paulddraper wrote: post #95

Earlier quoted context omitted.

Why do you have a limit on the length of a field? So it can fit in a database, i.e. with a certain number of bytes?

Without a limit on password length, an attacker can DOS you by forcing you to run your KDF on gigabyte-sized strings.

They're only denying service to themselves if you run the KDF locally.

Re: Emoji.length == 2

#137

Post by geocar »

gtrubetskoy wrote: post #93

In Go: func main() { shit := "\U0001f4a9" fmt.Printf("len of %s is %d\n", shit, utf8.RuneCountInString(shit)) } $ len of � is 1 Though I can't say that this is all that intuitive either...

Codepoints still aren't the same as characters.

Consider the examples given about combining emoji; Consider two runes that make one character: e and ◌́

Re: Emoji.length == 2

#138

Post by Manishearth »

carapace wrote: post #83
phkahler wrote: post #2

Unicode is fucked. All these bullshit emojis remind me of the 1980s when ASCII was 7 bits but every computer manufacturer (Atari, Commodore, Apple, IBM, TI, etc...) made their own set of characters for the 128 values of a byte beyond ASCII. Of course Unicode is a global standard so your pile-of-poop emoji will still be a pile-of-poop on every device even if the amount of steam is different for some people. It's beyon…

Unicode is a conflation of two ideas, one good and the other impossible. The good idea is to have a standard mapping from numbers to little pictures (glyphs, symbols, kanji, ideograms, cuneiform pokings in dried clay, scratches on a rock, whatever.) This is really all ASCII was. The impossible idea is to encode human languages into bits. This can't be done and will only continue to cause heartache in those who try. A…

I hate this argument every time I see it because it's invariably used in the wrong place.

Yes, the goal of encoding all human languages into bits is one that's near impossible. Unicode tries, and has broken half-solutions in many places. Lots of heartache everywhere.

This is completely irrelevant to the discussion here. The issue of code points not always mapping to graphemes is only an issue because programmers ignore it. It's a completely solved problem, theoretically speaking. It's necessary to be able to handle many scripts, but it's not something that "breaks" unicode.

Re: Emoji.length == 2

#139

Post by xem »

Here are my 2 cents: you can decompose an Unicode string with the ES6 spread operator:

[..."(insert 5 poo emoji here)"].length === 5

[..."(insert 5 poo emoji here)"][1] === "(poo emoji)"

Re: Emoji.length == 2

#140

Post by paulddraper »

lucaspiller wrote: post #119
nurettin wrote: post #111

Earlier quoted context omitted.

How does Ruby struggle with unicode?

This is from a couple of weeks ago, there's a few things broken still, but what languages do have full support out of the box? http://blog.honeybadger.io/ruby-s-unicode-support/

Swift
Post reply on HN