(I should probably try this myself, but…) The equivalent programs on other systems are HarfBuzz (open-source) and Uniscribe / DirectWrite (Windows). Did you consider checking what they do with the equivalent text (presumably they don't crash)?
Picking Apart the Crashing iOS String
111–120 of 182 posts
Re: Picking Apart the Crashing iOS String
#112Not directly related to the crash, but I have a question about Telugu and similar scripts: How do their speakers think about the structure of the script? Do they consider each vowel and consonant a separate "thing" that just happens to get written as a complex grapheme, or is the grapheme the unit you think about and it just happens to be made up of smaller parts? I.e https://en.wikipedia.org/wiki/Telugu_script#Conso…
the sounds get "joined". the extensions to the vowels are added to consonants to create a compound. for example: letter "Ka" క becomes "Kaa" కా by adding "Ka" and "A" like A in apple. అ becomes the line and circle that gets added to the consonant. Consonants are not mixed with each other, cannot be. this seems like it could help for those interested in more info https://www.omniglot.com/writing/telugu.htm
I might be wrong or I am not getting your meaning but I do think consonants can be “mixed”.Say విక్రమ .But I think we are considering mixing differently.
Re: Picking Apart the Crashing iOS String
#113Earlier quoted context omitted.
The crash seems to be in CoreText. CoreText is embedded/linked in Messages, Spotlight, Springboard, etc. CoreText is written in C. The fix would be to rewrite CoreText in a memory safe language like Swift. This would be “hard”. Or put CoreText in an XPC container. This would both be “hard” and result in terrible performance. For more details on how hard C, memory management, systems programming, and operating system…
I'm guessing C is how they get the performance they need. Re-writing in Obj-C or Swift would likely have speed tradeoffs.
Obj-C is C, or more accurately a superset of it.
Re: Picking Apart the Crashing iOS String
#114Earlier quoted context omitted.
The crash seems to be in CoreText. CoreText is embedded/linked in Messages, Spotlight, Springboard, etc. CoreText is written in C. The fix would be to rewrite CoreText in a memory safe language like Swift. This would be “hard”. Or put CoreText in an XPC container. This would both be “hard” and result in terrible performance. For more details on how hard C, memory management, systems programming, and operating system…
I'm guessing C is how they get the performance they need. Re-writing in Obj-C or Swift would likely have speed tradeoffs.
Sure, but would it crash the entire OS when there’s a problem?
Re: Picking Apart the Crashing iOS String
#115@Manishearth
Thats a great explanation (though quite a bit of it was too much for me to understand).
The only correlation I can think of, is that the Kannada script keyboard and full support for the script rendering in the UI didn't exist in iOS until iOS 11.0. So it is crashing for Indic scripts that were supported pre iOS 11. Probably Apple tweaked something in iOS 11 for the languages already supported and broke something.
Re: Picking Apart the Crashing iOS String
#116Earlier quoted context omitted.
Why does everyone assume only this one specific character combination causes the issue? Generalists can't generalize, I guess. IOS/OSX are notoriously bad at unicode parsing... These have been problems for YEARS and there are many many known character combinations that cause these types of issues. You seriously think every character has unique code to process it? Of course not... there are general subsystems that pro…
I don't say "unicode" because I'm not certain it has to do specifically with unicode handling, and not a font stack bug (that would occur on other encodings). One of the previous crashes, for example, had nothing to do with unicode -- it had to do with the fact that some arabic strings get larger, visually, when you shorten them. Folks these days say "unicode" when talking about anything relevant to non-Latin text, w…
The problem is not with the characters themselves though, it's with how CoreText processes and parses the unicode symbols.
Don't get me wrong, I know fonts can be malicious, even... but given the history[0] here, with these unicode[1] issues... I think it's pretty safe to say the issue is not with a specific font, per-se, but with CoreText and unicode parsing. For example, when I ran OSX on my old macbook pro, I used open source fonts on my terminal, and this unicode parsing bug still happened.
In 2015 an Apple spokesperson had this to say: "We are aware of an iMessage issue caused by a specific series of unicode characters and we will make a fix available in a software update."
> BOM or endianness don't seem to be relevant to this bug.
Yet on your blog you allude to just that with the left/right comments, though, to be fair, you state that you really don't know the problem:
> I don’t really have one guess as to what’s going on here – I’d love to see what people think – but my current guess is that the “affinity” of the virama to the left instead of the right confuses the algorithm that handles ZWNJs after viramas into thinking the ZWNJ applies to the virama (it doesn’t, there’s a consonant in between), and this leads to some numbers not matching up and causing a buffer overflow or something.
This is claimed to be a dissection of the issue, but there is not even a stack trace present, and yet you joke about that...
> Yes, I could attach a debugger to the crashing process and investigate that instead, but that’s no fun
Nah, you should do that... and you'll likely see that it's CoreText being the same old piece of shit as usual. If it was a font problem, then loading that font on a different system that uses a different rendering engine should reproduce the same problem. It doesn't, I tried that years ago.
Sure, I would posit, that potentially, this is a whole different bug, but... given the history, and the repeated failed attempts to fix this entire class of issues... it's safe to say that IOS and OSX do not handle unicode very well.
[0] https://www.theregister.co.uk/2013/09/04/unicode_of_death_cr...
[1] https://www.theregister.co.uk/2015/05/27/text_message_unicod...
Re: Picking Apart the Crashing iOS String
#117Earlier quoted context omitted.
> PS: If a web page wants to crash, it easily do so by allocating memory in a loop, isn't this also a bug :) And probably want we should fix.. I think all browsers are susceptible, though last I tried in Firefox one had to not just allocate but also fill the memory with garbage.
He means crash the single web page doing the loop... how would you fix this? Isn't the correct behavior when an application goes past resource limits to crash the app? It doesn't crash the browser or any other open tabs.
Better process management and limitations between tabs on browsers can help.
For instance, this site will crash most Firefox browsers but not Chrome because they limit the modal dialog rates: fan-pages[dot]herokuapp[dot]com (be warned that it may crash your browser even with JS disabled).
Re: Picking Apart the Crashing iOS String
#118I cannot trigger this crash on iOS 10.3.2, despite repeated attempts. I can reliably crash friends' phones by iMessage for all friends on iOS 11. That suggests to me that Apple made changes to CoreText, and did not perform adequate regression testing. All software has bugs. I understand that. But I suspect a large part of power users' and developers' growing frustration with Apple is that they keep introducing severe…
> Honestly, how do you not have a stringent regression testing requirement for changes to the "Core" of the operating system?
By not managing expectations and deadlines properly. Especially if schedules are overriden from a manager who is not aware of the ramifications of changing something in the core.
Re: Picking Apart the Crashing iOS String
#119Does anyone have a link to a thorough reverse-engineering of the bug from someone decompiling/source-diving?
I tried replicating it on High Sierra out of curiosity (single text file with the broken character, opened in TextEdit to cause a crash) and noticed that the crash doesn't happen in consistently the same way each time (I got 8 crashes in a calloc call in CoreText's OTL::CoverageBitmap::Reset, 4 crashes in a malloc in Foundation with no CoreText in the stack trace, plus 10 KERN_INVALID_ADDRESS errors from various places, as well as one time it opened successfully with no immediate problems). Seems like a memory corruption error somewhere earlier, but I'm not sure how to figure out where–I tried rooting around in CoreText with Hopper for a bit but didn't get very far.
Re: Picking Apart the Crashing iOS String
#120The broader question is, I think, how can any textual sequence cause a crash. Text should be regarded very skeptically by the text renderer. What could this be doing that it invokes a crash? Where else is this team not being careful? It reeks of process problems.
Latin text gives people a misleading idea as to how simple text is. For Latin, each character is generally an independent unit with independent metrics isolated from its environment, with a small set of exceptions to this rule (ligatures). East Asian ideographs bring up interesting questions about what constitutes a character, with Unicode "solving" the problem by saying "every distinct rendering is a distinct charac…
One great way of dealing with this at a software engineering level is to stop operating on arrays of characters, and instead add functionality on a case-by-case basis.
For example, you have a wrapper that only lets you iterate forwards. Once you hit languages where you need to "move back", you have to explicitly code in the functioanality. And in theory you can do it in a safe way.
Sure, some might argue you'll end back at arrays. But I believe that if you encode the traversals in a specific way, you'll at least end up at bounds-safe arrays.
For example, if you use an array as a queue, but some other part of the code doesn't, you're gonna have problems. But if you wrap your thing as a queue, no other part of the code will be able to pierce the veil.
Though I don't know how how well C abstractions let you do this.