Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WikiReader

en.wikipedia.org

Post reply on HN
Print view View original

11–20 of 27 posts

Re: WikiReader

#11

Post by peterburkimsher »

tech234a wrote: post #2

Although not a dedicated hardware device, Kiwix makes it very easy to browse Wikipedia offline on both desktop and mobile devices: https://kiwix.org/en/applications/

Kiwix is good, but the file size of offline Wikipedia is bloated compared to Wiki2Touch.

Kiwix is over 100 GB for the English Wikipedia. https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/

wikipedia_en_all_maxi_2023-10.zim 31-Oct-2023 07:37 103214717026

http://www.haukap.com/wiki2touch/

Wiki2Touch is about 9 GB for en-wiki as of 2012, and about 14 GB now.

I've been working on a Wiki2Touch reader for more modern iPhones. I've got the BZ2 decompression working, now I'm just cleaning up the markup to HTML parser in JavaScript.

If you're interested, please email me and I'll send you the code! You can install it for a week on a non-jailbroken device using XCode self-signing, and if you're jailbroken you can install the Immortal tweak to make it work forever.

If you have a developer subscription and want to put it on the App Store, I'm be very happy to let you do that. I just don't want to pay $100 for the privilege of running my own code.

Re: WikiReader

#12

Post by tdeck »

SushiHippie wrote: post #8
kohbo wrote: post #7

Earlier quoted context omitted.

Sad to see the Android app hasn't been updated in 5 years

https://play.google.com/store/apps/details?id=org.kiwix.kiwi... > Updated on Oct 22, 2021 https://github.com/kiwix/kiwix-android last commit 9 hours ago EDIT: Oh I see the F-Droid one wasn't updated in a while https://github.com/kiwix/kiwix-android/issues/3376 https://github.com/kiwix/kiwix-android/issues/3460

The play store tells me it won't run on my phone because it was made for an older version of Android.

Re: WikiReader

#14

Post by khimaros »

peterburkimsher wrote: post #11
tech234a wrote: post #2

Although not a dedicated hardware device, Kiwix makes it very easy to browse Wikipedia offline on both desktop and mobile devices: https://kiwix.org/en/applications/

Kiwix is good, but the file size of offline Wikipedia is bloated compared to Wiki2Touch. Kiwix is over 100 GB for the English Wikipedia. https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/ wikipedia_en_all_maxi_2023-10.zim 31-Oct-2023 07:37 103214717026 http://www.haukap.com/wiki2touch/ Wiki2Touch is about 9 GB for en-wiki as of 2012, and about 14 GB now. I've been working on a Wiki2Touch reader for more modern iP…

what accounts for the large difference in size?

Re: WikiReader

#16

Post by peterburkimsher »

khimaros wrote: post #14
peterburkimsher wrote: post #11

Earlier quoted context omitted.

Kiwix is good, but the file size of offline Wikipedia is bloated compared to Wiki2Touch. Kiwix is over 100 GB for the English Wikipedia. https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/ wikipedia_en_all_maxi_2023-10.zim 31-Oct-2023 07:37 103214717026 http://www.haukap.com/wiki2touch/ Wiki2Touch is about 9 GB for en-wiki as of 2012, and about 14 GB now. I've been working on a Wiki2Touch reader for more modern iP…

what accounts for the large difference in size?

I think Kiwix includes images.

Re: WikiReader

#17

Post by lxgr »

peterburkimsher wrote: post #16
khimaros wrote: post #14

Earlier quoted context omitted.

what accounts for the large difference in size?

I think Kiwix includes images.

Even the non-image English file is about 60GB, so I’m really curious what Wiki2Touch does differently!

What date is that 14GB version from? And is it really not filtered at all?

Re: WikiReader

#18

Post by peterburkimsher »

lxgr wrote: post #17
peterburkimsher wrote: post #16

Earlier quoted context omitted.

I think Kiwix includes images.

Even the non-image English file is about 60GB, so I’m really curious what Wiki2Touch does differently! What date is that 14GB version from? And is it really not filtered at all?

The 14GB is from 2021-03-01. It's not filtered, it just stores the wikitext and an index of the article names, compressed with BZ2.

The XML, which is larger, is now 21 GB. How did you get 60GB for the non-image English file?

https://dumps.wikimedia.org/enwiki/latest/

enwiki-latest-pages-articles.xml.bz2 02-Dec-2023 02:38 21557219519

Re: WikiReader

#19

Post by lxgr »

peterburkimsher wrote: post #18
lxgr wrote: post #17

Earlier quoted context omitted.

Even the non-image English file is about 60GB, so I’m really curious what Wiki2Touch does differently! What date is that 14GB version from? And is it really not filtered at all?

The 14GB is from 2021-03-01. It's not filtered, it just stores the wikitext and an index of the article names, compressed with BZ2. The XML, which is larger, is now 21 GB. How did you get 60GB for the non-image English file? https://dumps.wikimedia.org/enwiki/latest/ enwiki-latest-pages-articles.xml.bz2 02-Dec-2023 02:38 21557219519

I meant the Kiwix dump (https://download.kiwix.org/zim/wikipedia_en_all_nopic.zim – careful, 60GB!).

At a first glance, the Wikimedia XML dump does not look substantially different from what Kiwix/ZIM does with compressed HTML: They're both compressed (bz2 for the Wikimedia dump, zstd or LZMA for Kiwix/ZIM), and both compress multiple files at once, so inter-file redundancy should hopefully be significantly reduced.

HTML seems a bit more verbose than the Mediawiki syntax (plus the XML header for each article), but I'd be surprised if that actually accounted for a 3x difference in size.

Then again, Kiwix seems to have experimented with shared dictionary brotli compression, which supposedly yields an >2x improvement: https://github.com/openzim/libzim/issues/144

I wonder if their current zstd implementation also uses shared dictionaries. If not, that might just be the reason: If ZIM compression chunks are much smaller than the bz2 streams of the Wikimedia dumps, there would still be a lot of redundancy between chunks.

Post reply on HN