Earlier quoted context omitted.
Any kinks would be appreciated.
I think you may have had a Freudian slip ;)
Indeed! Kinks still welcome. (was going to make a joke about fat-fingering, butt decided to let it lay).
Wayback Machine was down
11–20 of 34 posts
Re: Wayback Machine was down
#12Now would be a great time to donate to the Internet Archive if you're able to. They can surely use the help.
If only shareholders would think the same way about PG&E and other companies that could use infrastructure upgrades ...
Re: Wayback Machine was down
#13In the meantime, distributed archiving ftw, run your own archives with Webrecorder.io, ArchiveBox.io, SingleFile, kiwix.org, etc!
The kiwix Wikipedia-en full scrape with images has been broken for over 18 months, I think they could use some technical help. I tried running their scraper myself on a nice AWS instance and it just stalls after many days of downloading articles. Could probably use a rewrite. ;)
https://sourceforge.net/p/kiwix/discussion/604121/thread/1f2...
Re: Wayback Machine was down
#14Now would be a great time to donate to the Internet Archive if you're able to. They can surely use the help.
If only shareholders would think the same way about PG&E and other companies that could use infrastructure upgrades ...
Downvotes! Yep, this is exactly why we have crappy infrastructure.
Re: Wayback Machine was down
#15Re: Wayback Machine was down
#16In the meantime, distributed archiving ftw, run your own archives with Webrecorder.io, ArchiveBox.io, SingleFile, kiwix.org, etc!
The kiwix Wikipedia-en full scrape with images has been broken for over 18 months, I think they could use some technical help. I tried running their scraper myself on a nice AWS instance and it just stalls after many days of downloading articles. Could probably use a rewrite. ;) https://sourceforge.net/p/kiwix/discussion/604121/thread/1f2... https://github.com/openzim/mwoffliner/issues/1020 https://github.com/openzim…
I'm not sure what your use case is so maybe this isn't helpful, but Wikipedia has weekly or so database dumps that you can download, as well as static HTML (although that might be more out of date)
Re: Wayback Machine was down
#17Earlier quoted context omitted.
If only shareholders would think the same way about PG&E and other companies that could use infrastructure upgrades ...
Downvotes! Yep, this is exactly why we have crappy infrastructure.
FYI: downvotes because (0) off-topic, (a) not constructive, (b) the world is bigger than California and there’s a lot of great infrastructure elsewhere, (c) PG&E is probably about average, (d) complaining about downvotes without thinking about why people had reason to downvote you.
Re: Wayback Machine was down
#18In the meantime, distributed archiving ftw, run your own archives with Webrecorder.io, ArchiveBox.io, SingleFile, kiwix.org, etc!
The kiwix Wikipedia-en full scrape with images has been broken for over 18 months, I think they could use some technical help. I tried running their scraper myself on a nice AWS instance and it just stalls after many days of downloading articles. Could probably use a rewrite. ;) https://sourceforge.net/p/kiwix/discussion/604121/thread/1f2... https://github.com/openzim/mwoffliner/issues/1020 https://github.com/openzim…
The whole zim file infrastructure is pretty broken. I've been trying to put together a system for generating a WARC file by rendering all the wikitext content in a database dump, which is a lot more reasonable of an approach.
Rendering wikitext is challenging though, since wikitext can include chunks of other wikitext, and wikitext can use some pretty complicated templating functionality.
Oddly enough where I've run into the biggest issues is in weird slowdowns of the python WARCIO library that making dealing with large archives just about impossible. I haven't had time to really track that down, but if anyone want to it's pretty easy to reproduce, just try adding a few million lorum-ipsum articles and look at how far from linear time it's running.
There are a lot of advantages to starting from a dump, you can provide much better tools for filtering articles, probably even provide rudimentary document classification. You can also do things like re-compress and minify images, a dump intended for a cellphone probably doesn't need 4k images.
WARC is also probably a better tool for distributing web-archive type content, like wikipedia dumps. You can distribute a package of text content and image content as separate files, for example. Generally I have not been very impressed with the quality of ZIM file tooling. One disadvantage is you need to provide separate search indexing, but that's doable.
I'd love to be able to get a wikimedia grant to work on this, and take on less contract work, but so far their grant process is pretty hard to follow.
Re: Wayback Machine was down
#19Earlier quoted context omitted.
The kiwix Wikipedia-en full scrape with images has been broken for over 18 months, I think they could use some technical help. I tried running their scraper myself on a nice AWS instance and it just stalls after many days of downloading articles. Could probably use a rewrite. ;) https://sourceforge.net/p/kiwix/discussion/604121/thread/1f2... https://github.com/openzim/mwoffliner/issues/1020 https://github.com/openzim…
I'm not sure what your use case is so maybe this isn't helpful, but Wikipedia has weekly or so database dumps that you can download, as well as static HTML (although that might be more out of date) https://en.wikipedia.org/wiki/Wikipedia:Database_download
Kiwix has a mobile app that runs a local copy of full Wikipedia (~80GB) from the device, with formatting and pictures and everything. I'm not aware of any other fully-baked solution that does this.
It looks like the pictures are not available as a dump from Wikimedia; the page you link to implies that they are, but the latest dump is from 2012.