Live data from Hacker News

Reverse-Engineering Apple Dictionary (2020)

fmentzer.github.io

21–30 of 61 posts

Re: Reverse-Engineering Apple Dictionary (2020)

#21
Another way to extract all of those compressed zip files would have been to use binwalk.

    binwalk --dd='.*' *.asset
Edit: should have read the comments before I posted this, because enragedcacti already mentioned this tool an hour ago.

Re: Reverse-Engineering Apple Dictionary (2020)

#22
post #17

Earlier quoted context omitted.

Yes, a lot of people involved in dictionary processing are worried about copyright! After I wrote my post about Apple's dictionary files, I got a mysterious email showing up in my inbox. The email was from someone who's spent some time writing code to do the same thing, but doesn't want to post it under his own name in case he falls fowl of his country's DMCA equivalent. Crazy. He said I could post his code under the…

I have helped with the writing and editing of a number of dictionaries over the years. It's difficult, highly specialized work. For some of the jobs, the only compensation has been a share of the royalties from future sales. I doubt if many people like me are enthusiastic about the dictionary data becoming accessible for free.

Yes, dictionary content is the revenue source of dictionary vendors, so of course they don't want anyone to use it without permission. On the other hand there are more and more open-data projects (I started one myself), often based on printed dictionaries that felt in the public domain.

Re: Reverse-Engineering Apple Dictionary (2020)

#23

Thank you for posting this code on Github! There has been some reverse-engineering done on the language dictionaries bundled with Mac OS, and it's nice to know that the same model is being used on the Apple Watch! I look forward to seeing your dictionary app. https://josephg.com/blog/reverse-engineering-apple-dictionar... There's also a command-line tool that can query the dictionary: https://github.com/takumakei/osx…

>Apple's word segmentation

Unless they changed it, it's probably similar to CFStringTokenizer which used ICU Boundary Analysis (and maybe mecab for Japanese).

Re: Reverse-Engineering Apple Dictionary (2020)

#24
post #5

I assume one reason Apple has made it more challenging to extract the dictionary resources is in order to satisfy licensing constraints with the dictionary authors. I wonder if they'd block an app like this through the App Store submission process, if submitted.

I suspect it’s more likely they have a bunch of internal frameworks for creating, accessing, and distributing simple databases (something akin to SQLite). So the difficulty we’re seeing here isn’t deliberate obfuscation, but rather just an dev using a database structure to make app design and word lookup easier. If there really was licensing concerns I would expect there to at-least be some basic encryption, not just…

Apple use regular SQLite for a whole bunch of other applications, such as Apple Photos. I wonder why they didn't use it here?

Re: Reverse-Engineering Apple Dictionary (2020)

#25
This was an amazingly delightful read. I was hoping it would shed some light on a long-time project I had which was reversing the Oxford language dictionaries that were included on CDROM with the big printed texts. (I already did it, but with a debugger instead of by reversing the binary format). Alas, it did not, but it was super encouraging to see the enthusiasm and interest in language dictionaries.

Re: Reverse-Engineering Apple Dictionary (2020)

#26
post #24

Earlier quoted context omitted.

I suspect it’s more likely they have a bunch of internal frameworks for creating, accessing, and distributing simple databases (something akin to SQLite). So the difficulty we’re seeing here isn’t deliberate obfuscation, but rather just an dev using a database structure to make app design and word lookup easier. If there really was licensing concerns I would expect there to at-least be some basic encryption, not just…

Apple use regular SQLite for a whole bunch of other applications, such as Apple Photos. I wonder why they didn't use it here?

The dictionary app (and hence its database) may go all the way back to NeXTStep/Openstep.

Maybe not all of it. But I wouldn't doubt if bits of it went all the way back to the early days where a fancy dictionary was one of the star features of NeXTStep.

Re: Reverse-Engineering Apple Dictionary (2020)

#27
post #24

Earlier quoted context omitted.

I suspect it’s more likely they have a bunch of internal frameworks for creating, accessing, and distributing simple databases (something akin to SQLite). So the difficulty we’re seeing here isn’t deliberate obfuscation, but rather just an dev using a database structure to make app design and word lookup easier. If there really was licensing concerns I would expect there to at-least be some basic encryption, not just…

Apple use regular SQLite for a whole bunch of other applications, such as Apple Photos. I wonder why they didn't use it here?

The underlying file format and associated libraries are probably largely unchanged from the NeXTStep era. If it ain’t broke, don’t fix it.

http://toastytech.com/guis/ns20mail.png

(Edit: beaten by 30 seconds!)

Re: Reverse-Engineering Apple Dictionary (2020)

#28
post #17

Earlier quoted context omitted.

I have helped with the writing and editing of a number of dictionaries over the years. It's difficult, highly specialized work. For some of the jobs, the only compensation has been a share of the royalties from future sales. I doubt if many people like me are enthusiastic about the dictionary data becoming accessible for free.

Yes, dictionary content is the revenue source of dictionary vendors, so of course they don't want anyone to use it without permission. On the other hand there are more and more open-data projects (I started one myself), often based on printed dictionaries that felt in the public domain.

What’s the state of Wiktionary like in your opinion?

Re: Reverse-Engineering Apple Dictionary (2020)

#29
post #17

Earlier quoted context omitted.

I have helped with the writing and editing of a number of dictionaries over the years. It's difficult, highly specialized work. For some of the jobs, the only compensation has been a share of the royalties from future sales. I doubt if many people like me are enthusiastic about the dictionary data becoming accessible for free.

Yes, dictionary content is the revenue source of dictionary vendors, so of course they don't want anyone to use it without permission. On the other hand there are more and more open-data projects (I started one myself), often based on printed dictionaries that felt in the public domain.

I started one myself, too, more than twenty years ago, in a naive burst of enthusiasm about the potential of online collaboration. It was an attempt to create a new comprehensive Japanese-English dictionary from scratch, not based on existing dictionaries [1]. Other volunteers were equally enthusiastic, but the immensity of the task before us, and the competing time pressures of paying work, caused us to gradually stop working on it.

[1] https://t-nex.jp/dictionaries/jekai/index.html

Re: Reverse-Engineering Apple Dictionary (2020)

#30
Has anyone reversed-engineered the Apple emoji dictionary that maps some keywords to emojis? Last time I checked, they only shipped binaries on the newest MacOS. Would love to use that mapping to elevate search on my custom emoji picker.
Post reply on HN