Reverse-Engineering Apple Dictionary (2020) • Hacker News Board

Re: Reverse-Engineering Apple Dictionary (2020)

#51

Post by LinguaBrowse » Mon, Sep 13, 2021, 8:34 AM UTC

atorodius wrote: post #46

Author here, funny to see this popping up on HN :) Was definitly a fun ride making this.

Nice read!

Just to mention that Core Foundation includes tokenisers (CFStringTokenizer) and lemmatisers (NSLinguisticTagger) of its own in case you ever wanted to avoid the Python processing!

Re: Reverse-Engineering Apple Dictionary (2020)

#52

Post by bouk » Mon, Sep 13, 2021, 9:53 AM UTC

enragedcacti wrote: post #2

For anyone else needing to tackle something like this, its definitely worth checking out Binwalk [1]. It is meant for extracting firmware but it works decently well on most files-in-files type data formats. [1] https://github.com/ReFirmLabs/binwalk

This was my first instinct as well! Running binwalk on the binary dictionary file immediately outputs that it's a series of zlib-compressed values.

Re: Reverse-Engineering Apple Dictionary (2020)

#53

Post by diimdeep » Mon, Sep 13, 2021, 10:41 AM UTC

Unfortunately it is not possible to implement add-on(add source) to Dictionary.app that is dynamic, like built-in Wikipedia; for example to query urbandictionary.com; only static offline is possible. I tried to investigate this years ago, don't think something changed since then.

Re: Reverse-Engineering Apple Dictionary (2020)

#54

Post by bhl » Mon, Sep 13, 2021, 5:27 PM UTC

cprecioso wrote: post #49

bhl wrote: post #30

Has anyone reversed-engineered the Apple emoji dictionary that maps some keywords to emojis? Last time I checked, they only shipped binaries on the newest MacOS. Would love to use that mapping to elevate search on my custom emoji picker.

I think it’s just the name data from the Unicode standard? (possibly the CLDR data but I might be wrong)

It's more than just the CLDR keywords (https://unicode.org/emoji/charts/emoji-list.html), since text replacement or autocomplete shows up for words not on that list.

Re: Reverse-Engineering Apple Dictionary (2020)

#55

Post by bhl » Mon, Sep 13, 2021, 5:28 PM UTC

luigi23 wrote: post #36

bhl wrote: post #30

Has anyone reversed-engineered the Apple emoji dictionary that maps some keywords to emojis? Last time I checked, they only shipped binaries on the newest MacOS. Would love to use that mapping to elevate search on my custom emoji picker.

you may try assign text replacements for emojis: https://www.iphonehacks.com/2016/04/text-replacement-mac.htm...

I'm trying to find the source data Apple uses for text replacements / suggestions.

Re: Reverse-Engineering Apple Dictionary (2020)

#56

Post by solarmist » Fri, Sep 17, 2021, 11:11 PM UTC

tim-- wrote: post #21

Another way to extract all of those compressed zip files would have been to use binwalk. binwalk --dd='.*' *.asset Edit: should have read the comments before I posted this, because enragedcacti already mentioned this tool an hour ago.

Thank you I didn't know about this Binwalk.

I used it and was able to figure out the remaining bits of the file format thanks to you and other tips in this thread.

https://github.com/solarmist/apple-peeler

Re: Reverse-Engineering Apple Dictionary (2020)

#57

Post by solarmist » Fri, Sep 17, 2021, 11:12 PM UTC

cogburnd02 wrote: post #9

would be really really cool if someone could make a small script to convert these into a format understandable by dict:// https://en.wikipedia.org/wiki/DICT

I've created a first step towards that. A general purpose tool for extracting the data back into XML.

https://github.com/solarmist/apple-peeler

Re: Reverse-Engineering Apple Dictionary (2020)

#58

Post by solarmist » Fri, Sep 17, 2021, 11:15 PM UTC

atorodius wrote: post #46

Author here, funny to see this popping up on HN :) Was definitly a fun ride making this.

Thanks for sharing this.

I've used your article and other previous attempts to clear up the remaining unknown bits of the file format and build apple-peeler, a tool for extracting the XML from the dictionary files.

https://github.com/solarmist/apple-peeler

Re: Reverse-Engineering Apple Dictionary (2020)

#59

Post by cogburnd02 » Sun, Sep 19, 2021, 12:04 AM UTC

solarmist wrote: post #57

cogburnd02 wrote: post #9

would be really really cool if someone could make a small script to convert these into a format understandable by dict:// https://en.wikipedia.org/wiki/DICT

I've created a first step towards that. A general purpose tool for extracting the data back into XML. https://github.com/solarmist/apple-peeler

I bet it'd be pretty easy to make an awk script to convert XML into something like dictfmt(1)'s -j Jargon File format text.

https://linux.die.net/man/1/dictfmt http://www.catb.org/jargon/oldversions/jarg447.txt

Re: Reverse-Engineering Apple Dictionary (2020)

#60

Post by solarmist » Tue, Sep 21, 2021, 6:59 AM UTC

cogburnd02 wrote: post #59

solarmist wrote: post #57

Earlier quoted context omitted.

I've created a first step towards that. A general purpose tool for extracting the data back into XML. https://github.com/solarmist/apple-peeler

I bet it'd be pretty easy to make an awk script to convert XML into something like dictfmt(1)'s -j Jargon File format text. https://linux.die.net/man/1/dictfmt http://www.catb.org/jargon/oldversions/jarg447.txt

Maybe. Don't know. I've never really used dictfmt or the like.

What's it good for?