Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emoji.length == 2

blog.jonnew.com

Post reply on HN
Print view View original

121–130 of 166 posts

Re: Emoji.length == 2

#121

Post by kmill »

toast0 wrote: post #115
paulddraper wrote: post #95

Earlier quoted context omitted.

Why do you have a limit on the length of a field? So it can fit in a database, i.e. with a certain number of bytes?

Some limits are technical (and in that case the hard limit is often bytes, but sometimes code units or code points, or broken if you told MySQL utf8 instead of bytes or utf8mb4), but in many cases, the limits are for aesthetic purposes: a post title or a username often is often required to be fairly short to look nice; in an ascii or latin1 world, those limits are usually expressed in terms of characters, but graphem…

"Your username must be 1-4cm when printed with 12pt Times New Roman."

I kind of like the idea of minimum length in cm as a password requirement.

Re: Emoji.length == 2

#122

Post by slobotron »

lucaspiller wrote: post #119
nurettin wrote: post #111

Earlier quoted context omitted.

How does Ruby struggle with unicode?

This is from a couple of weeks ago, there's a few things broken still, but what languages do have full support out of the box? http://blog.honeybadger.io/ruby-s-unicode-support/

Perl6 https://perl6advent.wordpress.com/2015/12/07/day-7-unicode-p...

Re: Emoji.length == 2

#124

Post by rspeer »

Manishearth wrote: post #92
rspeer wrote: post #85

Earlier quoted context omitted.

I'm curious. Are interlinear ruby annotation codepoints actually used for their intended purpose anywhere? And what are you supposed to do when you encounter one? I know that they appear on Unicode's shitlist in [UTR#20], a proposed tech report that contained a table of codepoints that should not be used in text meant for public consumption. UTR#20 suggested things you could do when you encounter these codepoints, bu…

> Are interlinear ruby annotation codepoints actually used for their intended purpose anywhere? Yes > And what are you supposed to do when you encounter one? Nothing. Don't display them, or display some symbolic representation. You probably shouldn't make ruby happen here; if your text is intended to be rendered correctly use a markup language. ---------- Unicode is ultimately a system for describing text. Not all st…

Can you give an example of text in the wild that uses interlinear ruby annotation codepoints? Because I searched the Common Crawl for them, and every occurrence of U+FFF9 through U+FFFB seems to have been an accident that has nothing to do with Japanese.

Note that I didn't actually ask you about rendering.

I care from the point of view of the base level of natural language processing. Some decisions that have nothing to do with rendering are:

- Do they count as graphemes?

- What do you do when you feed text containing ruby characters to a Japanese word segmenter (which is not going to be okay with crazy Unicode control characters, even those intended for Japanese)?

- Could they appear in the middle of a phrase you would reasonably search for? Should that phrase then be searchable without the ruby? Should the contents of the ruby also be searchable?

Seeing how ruby codepoints are actually used would help to decide how to process them. But as far as I can tell, they're not actually used (markup is used instead, quite reasonably). So I'm surprised that your answer is a flat "Yes".

Re: Emoji.length == 2

#125

Post by Immortalin »

lucaspiller wrote: post #119
nurettin wrote: post #111

Earlier quoted context omitted.

How does Ruby struggle with unicode?

This is from a couple of weeks ago, there's a few things broken still, but what languages do have full support out of the box? http://blog.honeybadger.io/ruby-s-unicode-support/

Go

Re: Emoji.length == 2

#126

Post by libeclipse »

deathanatos wrote: post #76
danbruc wrote: post #5

The Unicode standard describes in Annex 29 [1] how to properly split strings into grapheme clusters. And here [2] is a JavaScript implementation. This is a solved problem. [1] http://www.unicode.org/reports/tr29/ [2] https://github.com/orling/grapheme-splitter

> This is a solved problem. Not … really. Yes, we "know" the solution, but the terrible APIs that compose so many language's standard string type goads the programmer into choosing the wrong method or type. JavaScript has — to an extent — the excuse of age. But the language still really (to my knowledge) lacks an effective way to deal with text that doesn't involve dragging in third-party libraries. You are not a hig…

I didn't know perfect unicode support in the stdlib was a requirement for being a high-level language.

Re: Emoji.length == 2

#127

Post by jcranmer »

paulddraper wrote: post #94
deathanatos wrote: post #76

Earlier quoted context omitted.

> This is a solved problem. Not … really. Yes, we "know" the solution, but the terrible APIs that compose so many language's standard string type goads the programmer into choosing the wrong method or type. JavaScript has — to an extent — the excuse of age. But the language still really (to my knowledge) lacks an effective way to deal with text that doesn't involve dragging in third-party libraries. You are not a hig…

> You are not a high-level language if your standard library struggles with Unicode So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages. HN teaches me something new every day.

What would you say Java is missing? Sure, it does have the "oops, we implemented Unicode when they said we only needed 16 bits problem" but unlike, say, JS, it actually handles astral plane characters well (e.g., the regex implementation actually says that . matches an astral plane code point rather than half of one).

It does have all the major Unicode annexes--normalization (java.text.Normalizer), grapheme clusters (java.text.BreakIterator), BIDI (java.text.Bidi), line breaking (java.text.BreakIterator), not to mention the Unicode script and character class tables (java.lang.Character). And, since Java 8, it does have a proper code point iterator over character sequences.

Re: Emoji.length == 2

#128

Post by jcranmer »

rspeer wrote: post #85
Manishearth wrote: post #43

Earlier quoted context omitted.

They're not supposed to render, it's purely for describing text. As are the interlinear ruby annotations.

I'm curious. Are interlinear ruby annotation codepoints actually used for their intended purpose anywhere? And what are you supposed to do when you encounter one? I know that they appear on Unicode's shitlist in [UTR#20], a proposed tech report that contained a table of codepoints that should not be used in text meant for public consumption. UTR#20 suggested things you could do when you encounter these codepoints, bu…

You could just read PropLists.txt to find the list of characters with the Deprecated property:

    0149          ; Deprecated # L&       LATIN SMALL LETTER N PRECEDED BY APOSTROPHE
    0673          ; Deprecated # Lo       ARABIC LETTER ALEF WITH WAVY HAMZA BELOW
    0F77          ; Deprecated # Mn       TIBETAN VOWEL SIGN VOCALIC RR
    0F79          ; Deprecated # Mn       TIBETAN VOWEL SIGN VOCALIC LL
    17A3..17A4    ; Deprecated # Lo   [2] KHMER INDEPENDENT VOWEL QAQ..KHMER INDEPENDENT VOWEL QAA
    206A..206F    ; Deprecated # Cf   [6] INHIBIT SYMMETRIC SWAPPING..NOMINAL DIGIT SHAPES
    2329          ; Deprecated # Ps       LEFT-POINTING ANGLE BRACKET
    232A          ; Deprecated # Pe       RIGHT-POINTING ANGLE BRACKET
    E0001         ; Deprecated # Cf       LANGUAGE TAG
(note that the ruby annotation codepoints aren't on that list).

The use in XML/HTML is no longer maintained by Unicode, it is maintained by the W3C instead: https://www.w3.org/TR/unicode-xml/.

Re: Emoji.length == 2

#130

Post by [deleted] »

xnyhps wrote: post #118
tomjakubowski wrote: post #100

Earlier quoted context omitted.

> A reasonable test for passwords is to run them through an IDNA checker, which checks whether a string is acceptable as a domain name component. This catches most weird stuff, such as mixed left-to-right and right-to-left symbols, zero-width markers, homoglyphs, and emoji. Why test this at all? It's not as if a website should ever need to render a user's password as text. Is there another use case for excluding this…

Suppose I include 'ü': LATIN SMALL LETTER U WITH DIAERESIS in my password. I switch to a different browser/OS/language and now when I enter "ü" I get 'u': LATIN SMALL LETTER U + ' ̈': COMBINING DIAERESIS. I can't log in anymore, though what I do is identical and defined to be equivalent. Especially if the password is hashed before comparing it, you can't treat it as just a sequence of bytes. You don't need to use IDN…

[deleted]
Post reply on HN