Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Picking Apart the Crashing iOS String

manishearth.github.io

Post reply on HN
Print view View original

11–20 of 182 posts

Re: Picking Apart the Crashing iOS String

#11

Post by Diggsey »

My guess would be that it's some aspect of measuring the text that is causing the crash: when you click in an editable text box, there is code to track down where the cursor should be placed. This is done by measuring various sub-strings of the whole line.

If measuring the sub-strings gives surprising results (sub-strings being visibly longer for example), this could cause the algorithm to fail in any number of interesting ways: for example if a binary search is used to locate the cursor position, it could break the invariants of the binary search.

Re: Picking Apart the Crashing iOS String

#13

Post by jfk13 »

nikanj wrote: post #10
spaceywilly wrote: post #5

Earlier quoted context omitted.

My question was, why was there not a unit test for this? It seems like it would be trivial to step through every character combination for each language they support and make sure it doesn't cause a crash.

Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112. This crashing sequence is five characters long. Your unit tests would have to go through 1.71650179e30 sequences to be guaranteed to catch this one. At a test rate of 1 millisecond per sequence, that's just 4×10^9 × the age of the universe, according to wolfram alpha.

Rendering Indic scripts with AAT fonts involves a series of finite state machines that are stored in the individual font. So don't forget to multiply by the number of different fonts that each need to be tested.

Re: Picking Apart the Crashing iOS String

#14

Post by ancarda »

nikanj wrote: post #10
spaceywilly wrote: post #5

Earlier quoted context omitted.

My question was, why was there not a unit test for this? It seems like it would be trivial to step through every character combination for each language they support and make sure it doesn't cause a crash.

Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112. This crashing sequence is five characters long. Your unit tests would have to go through 1.71650179e30 sequences to be guaranteed to catch this one. At a test rate of 1 millisecond per sequence, that's just 4×10^9 × the age of the universe, according to wolfram alpha.

Do you know how was this bug found?

Are there just enough people using iOS that these sorts of bugs can be found by mistake, or is someone fuzzing CoreText? Perhaps that can be applied to provide some kind of test coverage? Even if it’s not complete?

Re: Picking Apart the Crashing iOS String

#15

Post by pokpokpok »

nikanj wrote: post #10
spaceywilly wrote: post #5

Earlier quoted context omitted.

My question was, why was there not a unit test for this? It seems like it would be trivial to step through every character combination for each language they support and make sure it doesn't cause a crash.

Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112. This crashing sequence is five characters long. Your unit tests would have to go through 1.71650179e30 sequences to be guaranteed to catch this one. At a test rate of 1 millisecond per sequence, that's just 4×10^9 × the age of the universe, according to wolfram alpha.

true, but characters could be mapped to equivalence categories according to logic in the code

Re: Picking Apart the Crashing iOS String

#16

Post by jay-anderson »

brian-armstrong wrote: post #9
throwawayfinal wrote: post #6

Earlier quoted context omitted.

displaying language is full of corner cases. imagine having to render both english, japanese, and arabic in the same code paths, and inline with each other text display code is not so simple as it seems, and don't fall into the trap of "how hard could it really be"

I'm not saying it isn't hard - I acknowledge that getting it right is a huge task. But the cost of failure should be rendering incorrectly, not crashing or memory corruption.

There was a similar bug about a year ago as well. I'm surprised more fuzzing, static analysis, and general testing hasn't gone into this area since there have been issues found there already. Maybe they have done this extra work and just didn't catch this one. It's hard to know. Still feels like the way it fails is unacceptable.

Re: Picking Apart the Crashing iOS String

#17

Post by bitwize »

Manishearth wrote: post #3
donarb wrote: post #2

Very interesting article. It gives a bit more perspective to the problem than the simplistic view that iOS has a problem displaying a certain character onscreen. This particular bug shows the problem with multibyte characters (terminology may be wrong, perhaps multi glyph is better) where certain parts of the character become left or right associative based on context.

You want "multi-codepoint graphemes" probably. Or really "things involving combining characters". There isn't really a term for this.

I believe the Unicode term is "grapheme cluster".

Re: Picking Apart the Crashing iOS String

#18

Post by jcranmer »

brian-armstrong wrote: post #4

The broader question is, I think, how can any textual sequence cause a crash. Text should be regarded very skeptically by the text renderer. What could this be doing that it invokes a crash? Where else is this team not being careful? It reeks of process problems.

Latin text gives people a misleading idea as to how simple text is. For Latin, each character is generally an independent unit with independent metrics isolated from its environment, with a small set of exceptions to this rule (ligatures).

East Asian ideographs bring up interesting questions about what constitutes a character, with Unicode "solving" the problem by saying "every distinct rendering is a distinct character," necessitating somewhere in the region of 80,000 characters or so once all of them get added. Even more difficult are scripts like Korean Hangul or Egyptian and Mayan glyphs, which are composed from a relatively small set of independent units but laid out in blocks and sub-blocks themselves composed in linear text. Unicode has both precomposed Hangul characters and the individual Jamo radicals, but it has currently punted on being able to accurately Egyptian or Mayan text in the first place (although they do appear to be revisiting that decision).

Scripts like Arabic differ from Latin in that glyphs changing shape according to their surrounding context is the norm rather than the exception. However, in Arabic, you largely break this down into an initial/medial/final form, with some characters inducing a shift from medial back to initial. Indic scripts go far beyond this by needing to treat entire consonant clusters as single rendered glyphs.

The end result is that it's very easy to find that invariants that one expects to exist if one is used to Latin text to be violated in other languages. Some concepts of text metrics might not even exist in the first place. As speculated elsewhere, it's probable that the text is crashing because someone insufficiently versed in scripts is asserting an invariant that doesn't actually exist. It does not appear to be a rendering issue, but rather a slightly higher level operation on top of that instead.

Finding these sorts of issues pretty much requires extensive fuzzing with known problematic scenarios. The fault is caused by "I didn't know this exists" which is both a very reasonable situation (few UI experts are well-versed in the complexities of foreign scripts) and very hard to solve from a process perspective.

Re: Picking Apart the Crashing iOS String

#19

Post by perkee »

ancarda wrote: post #14
nikanj wrote: post #10

Earlier quoted context omitted.

Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112. This crashing sequence is five characters long. Your unit tests would have to go through 1.71650179e30 sequences to be guaranteed to catch this one. At a test rate of 1 millisecond per sequence, that's just 4×10^9 × the age of the universe, according to wolfram alpha.

Do you know how was this bug found? Are there just enough people using iOS that these sorts of bugs can be found by mistake, or is someone fuzzing CoreText? Perhaps that can be applied to provide some kind of test coverage? Even if it’s not complete?

This sequence begins the Telugu word for "knowledge" so maybe someone texted that to someone and it went viral from there. This is, of course, only speculation.

Re: Picking Apart the Crashing iOS String

#20

Post by goalieca »

Diggsey wrote: post #11

My guess would be that it's some aspect of measuring the text that is causing the crash: when you click in an editable text box, there is code to track down where the cursor should be placed. This is done by measuring various sub-strings of the whole line. If measuring the sub-strings gives surprising results (sub-strings being visibly longer for example), this could cause the algorithm to fail in any number of inter…

I believe the renderer is a brand new metal 2 for high sierra. The crashing 3rd party apps for iOS are using this new rendering too?
Post reply on HN