Earlier quoted context omitted.
The kiwix Wikipedia-en full scrape with images has been broken for over 18 months, I think they could use some technical help. I tried running their scraper myself on a nice AWS instance and it just stalls after many days of downloading articles. Could probably use a rewrite. ;) https://sourceforge.net/p/kiwix/discussion/604121/thread/1f2... https://github.com/openzim/mwoffliner/issues/1020 https://github.com/openzim…
The whole zim file infrastructure is pretty broken. I've been trying to put together a system for generating a WARC file by rendering all the wikitext content in a database dump, which is a lot more reasonable of an approach. Rendering wikitext is challenging though, since wikitext can include chunks of other wikitext, and wikitext can use some pretty complicated templating functionality. Oddly enough where I've run…
In general, I'd say that ZIM and WARC are not really direct competitors or solutions to the same problems, they're really for distinct use-cases. ZIM is a highly-compressed format that's designed solely for static articles and flat content, it doesn't really store headers or anything else that WARC does in order to support full request/response replaying. ZIM is optimized for storing thousands to millions of pages of homogenous content, WARC is optimized for high-fidelity collections of smaller amounts of content.
If you want to help out with our efforts, feel free to DM me on Twitter @theSquashSH or reply here and I can introduce you to the ZIM people (who get grants to improve this process on the regular, and are open to hiring contract workers).