Neat project, I like the idea. One thing from a quick read: you launch Chrome with --no-sandbox. Is there a good reason for that? Security wise it's probably not a good idea. If there is no reason, I'd suggest leaving the sandbox on! In any case, cool stuff :)
--no-sandbox is needed in docker, maybe they assume it will mostly run in docker?
Exactly. For downloading, Kage requires Chrome or Chromium. Running it inside Docker makes setup easier and keeps cleanup simple:
I've accumulated a bunch of old website archives over the years. The funny thing is the ugly HTML dumps have been more useful than the "perfect" archive. It's one of the reasons I've become a bigger fan of RSS over time. A feed from 10-ish years ago is often more usable today than a carefully preserved (application) website.
I have a project for creating and archiving RSS feeds, keeping the full history from the time the crawler starts. I need to clean up a bit, then will open source it soon.
Cool project! I know it's written in go, but it would be cool to see something like this which uses Cosmopolitan Libc + redbean or something similar to create a binary which runs anywhere. Would be fun to be able to pass around self-executable website archives. https://github.com/jart/cosmopolitan https://justine.lol/cosmopolitan/index.html https://redbean.dev (Certificates just expired for justine's website, just ig…
This could be a nice code golf project. It only needs a webview, a ZIM reader, and a way to append data to an existing binary and read it back.
I did something like that a very long time ago (Of course, I have forgotten)
Since when? You won't be able to make HTTP requests to localhost, as it'd be a different Origin, but I don't think any mainstream browser blocks JS outright when you use file:// to load and view HTML files.
Somewhere around 2019, each document loaded from file:// became its own origin in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1500453 (I didn't check when this happened in Chromium) Related WHATWG discussion: https://github.com/whatwg/html/issues/3099
Yeah, but that's fine, the document is .html, and it can load ./app.js or ./style.css just fine even if loaded by file:// (as long as it isn't initiated by JS itself, then Origin starts to matter a lot more), otherwise basically every single local HTML file would suddenly be broken, I don't think anyone would have accepted that even with the origin changes.
I was intrigued to see how the demo GIF in the README was generated: https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it: ascii-gif render docs/demo/kage.tape -o docs/static/dem…
Have you heard the good news about the terminal savior asciinema -- https://asciinema.org/
It's a cool tool/platform, but very different. Asciinema tries to make the "multimedia" itself better by making it actual text instead of being video/images, while the CLI command above turns actual text into multimedia supported by platforms already. Both are useful, both have their use cases :)
Submitting this to Hacker News is the right place! Thanks for your idea. I will consider implementing that :) Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.
I think the zim flow was perfect for offline use. I know I will be making use of it as soon as I can figure out how to pass chrome the cookies so I can be signed into the site. Didn't see it in the page, but I didn't look closely yet.
Not yet supporting cookies, since I created this tool for shadowing public websites first. I will add options to pass cookies later. It will pass them to the underlying Chrome/Chromium process, so it should not be hard to do.
But will look into this now, see if we can swap some stuff out. We’ve really liked the idea of an offline mirror, makes a lot of collaboration use cases simpler
I find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli
It seems this repo only saves one web page? What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.
> For example, all essays from paulgraham.com
Not the same thing, but I made a clone of pg’s website which can be used for exactly that: https://github.com/shawwn/pg
I tried to clone a HTTP (not HTTPS) site, and it's giving me `navigation failed: net::ERR_NAME_NOT_RESOLVED`. Even when I explicitly included the protocol with `http://`.