Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emoji.length == 2

blog.jonnew.com

Post reply on HN
Print view View original

111–120 of 166 posts

Re: Emoji.length == 2

#111

Post by nurettin »

paulddraper wrote: post #94
deathanatos wrote: post #76

Earlier quoted context omitted.

> This is a solved problem. Not … really. Yes, we "know" the solution, but the terrible APIs that compose so many language's standard string type goads the programmer into choosing the wrong method or type. JavaScript has — to an extent — the excuse of age. But the language still really (to my knowledge) lacks an effective way to deal with text that doesn't involve dragging in third-party libraries. You are not a hig…

> You are not a high-level language if your standard library struggles with Unicode So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages. HN teaches me something new every day.

How does Ruby struggle with unicode?

Re: Emoji.length == 2

#113

Post by rjbwork »

yongjik wrote: post #70
duskwuff wrote: post #13

Earlier quoted context omitted.

The one that still surprises me is Hangul (Korean script). Hangul characters are made of 24 basic characters (jamo) which represent consonant and vowel sounds, which are composed into Hangul characters representing syllables. Unicode has a block for Hangul jamo, but they aren't used in typical text. Instead, Hangul are presented using a massive 11K-codepoint block of every possible precomposed syllable. ¯\_(ツ)_/¯

I believe that was a necessary compromise to use Hangul on any software not authored by Koreans. "These are characters from a country you've never been to. Each three-byte sequence (assuming UTF-8) corresponds to a square-shaped character." --> Easy for everyone to understand, and less chance of screwup (as long as the software supports any Unicode at all). "These should be decomposed into sequences of two or three c…

Windows 10 latest chrome shows it properly, fwiw.

Re: Emoji.length == 2

#114

Post by kalleboo »

ge0rg wrote: post #48

I've gone through exactly the same discovery process when implementing faux stamps (something between images and Emoji) in my xmpp app yesterday. My idea was to increase the font size of a message that only consists of Emoji, depending on the number of Emoji in the message, like this: https://xmpp.pix-art.de/imagehost/display/file/2017-03-09_09... The code turned out more complex than first expected, mirroring the sa…

I'm working on a project that has to handle special rendering of emoji as well, and I simply ask the system "will this string render in the emoji font" and "how big of a rect do I need to render this string" to calculate the same thing, rather than trying to handle it myself and relying on assumptions about the sizing of the emoji. I figure this way I also future proof against whatever emoji they think up in the future.

Re: Emoji.length == 2

#115

Post by toast0 »

paulddraper wrote: post #95
toast0 wrote: post #66

Earlier quoted context omitted.

> Perhaps someone can shed light on a compelling use case for knowing the number of grapheme clusters in a particular string, because I haven't been able to think of one. If you have a limit on the length of a field, it helps to tell the user what it is in a way they understand. For non-technical users, bytes (and the embedded issue of encoding) and code points are both pretty esoteric, but number of symbols is less…

Why do you have a limit on the length of a field? So it can fit in a database, i.e. with a certain number of bytes?

Some limits are technical (and in that case the hard limit is often bytes, but sometimes code units or code points, or broken if you told MySQL utf8 instead of bytes or utf8mb4), but in many cases, the limits are for aesthetic purposes: a post title or a username often is often required to be fairly short to look nice; in an ascii or latin1 world, those limits are usually expressed in terms of characters, but graphemes might be the right thing to limit in a unicode world.

Re: Emoji.length == 2

#116

Post by toast0 »

fryguy wrote: post #82
TorKlingberg wrote: post #65

If you want to do Unicode correctly, you shouldn't ask for the "length" of a string. The is no true definition of length. If want to know how many bytes it uses in storage, ask for that. If you want to know how wide it will be on the screen, ask for that. Do not iterate over strings character by character.

How many dots/stars should one display for a password? That's a question that can't be answered by your two valid question. Are you suggesting that dots/stars shouldn't be displayed for passwords, since you can't ask how many "characters" it is?

If you have an entry indicator, it should probably be about the same width as the entered text; or if you're concerned about leaking precise length information for fields that aren't monospaced, you could add a dot each time the rendered text would increase in width.

Re: Emoji.length == 2

#117

Post by klodolph »

danbruc wrote: post #81
deathanatos wrote: post #76

Earlier quoted context omitted.

> This is a solved problem. Not … really. Yes, we "know" the solution, but the terrible APIs that compose so many language's standard string type goads the programmer into choosing the wrong method or type. JavaScript has — to an extent — the excuse of age. But the language still really (to my knowledge) lacks an effective way to deal with text that doesn't involve dragging in third-party libraries. You are not a hig…

That is what I meant, there is an existing algorithm to do this because the author tried to come up with one. That JavaScript fails to provide an implementation, well, too bad, but this is of course a problem one may have to solve in any language. And while other languages provide the necessary support at the language or standard library level, I would guess there are quite a few developers out there that are not eve…

It's not a problem that comes up that often, to be fair. Many of the cases where you think you'd need to split a string like that, you have some library doing the work for you. One of the main purposes is to figure out where valid cursor locations are in text editors... but in JavaScript, you just put a text field in your web page and let the browser do the heavy lifting. Same with text rendering... hand it off to a library which does the right thing.

Re: Emoji.length == 2

#118

Post by xnyhps »

tomjakubowski wrote: post #100
Animats wrote: post #24

Before emoji, fonts and colors were independent. Combining the two creates a mess. Try using emoji in an editor with syntax coloring. We got into this because some people thought that single-color emoji were racist.[1] So now there are five skin tone options. The no-option case is usually rendered as bright yellow, which comes from the old AOL client. They got it from the happy-face icon of the 1970s. Here's the curr…

> A reasonable test for passwords is to run them through an IDNA checker, which checks whether a string is acceptable as a domain name component. This catches most weird stuff, such as mixed left-to-right and right-to-left symbols, zero-width markers, homoglyphs, and emoji. Why test this at all? It's not as if a website should ever need to render a user's password as text. Is there another use case for excluding this…

Suppose I include 'ü': LATIN SMALL LETTER U WITH DIAERESIS in my password. I switch to a different browser/OS/language and now when I enter "ü" I get 'u': LATIN SMALL LETTER U + ' ̈': COMBINING DIAERESIS. I can't log in anymore, though what I do is identical and defined to be equivalent. Especially if the password is hashed before comparing it, you can't treat it as just a sequence of bytes.

You don't need to use IDNA for this, though. There are standards specifically for dealing with Unicode passwords, such as SASLprep (RFC 4013) and PRECIS (RFC 7564).

Re: Emoji.length == 2

#119

Post by lucaspiller »

nurettin wrote: post #111
paulddraper wrote: post #94

Earlier quoted context omitted.

> You are not a high-level language if your standard library struggles with Unicode So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages. HN teaches me something new every day.

How does Ruby struggle with unicode?

This is from a couple of weeks ago, there's a few things broken still, but what languages do have full support out of the box?

http://blog.honeybadger.io/ruby-s-unicode-support/

Re: Emoji.length == 2

#120

Post by slededit »

fryguy wrote: post #82
TorKlingberg wrote: post #65

If you want to do Unicode correctly, you shouldn't ask for the "length" of a string. The is no true definition of length. If want to know how many bytes it uses in storage, ask for that. If you want to know how wide it will be on the screen, ask for that. Do not iterate over strings character by character.

How many dots/stars should one display for a password? That's a question that can't be answered by your two valid question. Are you suggesting that dots/stars shouldn't be displayed for passwords, since you can't ask how many "characters" it is?

You could divide the length of the string by the length the '*' character in a monospaced font. It doesn't really make sense for a combining or other invisible character to get its own asterisk.
Post reply on HN