31–40 of 152 posts
I find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli
Earlier quoted context omitted.
That's neat! In my opinion, the WARC format is quite tricky and underspecified especially since HTTP2 introduced new semantics. It encodes too much in-band and requires rewriting of the server data. A mitmproxy capture is higher fidelity and supports capturing modern features such as WebSockets. I think if we could wrap Kage's crawler interactions by it and store its capture (the intercepted traffic), we could make a…
I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support. For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source)…
I find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli
It seems this repo only saves one web page? What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.
I think the misunderstanding stems from the browser's "Save As" reference in the description. It is misleading. You use "Save As" to save a single page, not an entire website.
Also, the description lacks a clear explanation of the project's purpose. It would be helpful to include a sentence explaining that the program downloads an entire website, not just a single page.
pandoc --from html --to epub --output /PATH/TO/FILE.epub https://example.comI find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli
And thanks for the link. Let me implement this single HTML feature, it looks nice to have!
That way, the page is self-contained as it is, but requires no bundled binary code to serve the site. It is actually safer security-wise.
The vendored script can be as simple as this:
const site = {
"path-1": " ... ",
"path-2": " ... ",
// More paths
}
function attachListeners() {
for (const [path, html] of Object.entries(site)) {
document.querySelector(`a[href=${path}]`).onclick = () => {
document.documentElement.outerHTML = html
attachListeners()
}
}
}
document.addEventListeners("DOMContentLoaded", attachListeners)If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like:
$ firefox $HOME/data/kage/paulgraham.com
Then the result would be useable on machines without kage nstalled.
> kage serve $HOME/data/kage/paulgraham.com If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like: $ firefox $HOME/data/kage/paulgraham.com Then the result would be useable on machines without kage nstalled.
Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif
The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it:
ascii-gif render docs/demo/kage.tape -o docs/static/demo.gif
Is the code also AI slop?
> kage serve $HOME/data/kage/paulgraham.com If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like: $ firefox $HOME/data/kage/paulgraham.com Then the result would be useable on machines without kage nstalled.
Usually JavaScript is blocked when you load pages that way.