Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kage – Shadow any website to a single binary for offline viewing

github.com

Post reply on HN
Print view View original

11–20 of 152 posts

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#11

Post by tamnd »

sanqui wrote: post #6

Cool concept. I would like to see this combined with mitmproxy for archive grade fidelity. You could be saving exactly the data served and at the same time a representation by a modern (contemporary) browser, with all JS having run. This combination would be my perfect replacement for the WARC format.

I'm working on WARC too, with format from Common Crawl!

By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#12

Post by sdevonoes »

tamnd wrote: post #3
maxloh wrote: post #2

I find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli

It seems this repo only saves one web page? What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.

[flagged]

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#13

Post by Dhavidh »

sanqui wrote: post #6

Cool concept. I would like to see this combined with mitmproxy for archive grade fidelity. You could be saving exactly the data served and at the same time a representation by a modern (contemporary) browser, with all JS having run. This combination would be my perfect replacement for the WARC format.

sound interesting

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#14

Post by tamnd »

wolttam wrote: post #10

One use I'd have for this is company wikis that you want to give folks easy offline access to (maybe the wiki has documentation that's useful at sites that don't have cellular coverage). Cool! It would be especially cool to have a version that didn't require the separate serving process - even though it's nifty you can package up a whole site as a single binary. Maybe a single HTML entrypoint shim with a bit of javas…

Submitting this to Hacker News is the right place! Thanks for your idea. I will consider implementing that :)

Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#15

Post by sanqui »

tamnd wrote: post #11
sanqui wrote: post #6

Cool concept. I would like to see this combined with mitmproxy for archive grade fidelity. You could be saving exactly the data served and at the same time a representation by a modern (contemporary) browser, with all JS having run. This combination would be my perfect replacement for the WARC format.

I'm working on WARC too, with format from Common Crawl! By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli

That's neat! In my opinion, the WARC format is quite tricky and underspecified especially since HTTP2 introduced new semantics. It encodes too much in-band and requires rewriting of the server data. A mitmproxy capture is higher fidelity and supports capturing modern features such as WebSockets. I think if we could wrap Kage's crawler interactions by it and store its capture (the intercepted traffic), we could make a potentially nice new archival format.

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#17

Post by tamnd »

sanqui wrote: post #15
tamnd wrote: post #11

Earlier quoted context omitted.

I'm working on WARC too, with format from Common Crawl! By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli

That's neat! In my opinion, the WARC format is quite tricky and underspecified especially since HTTP2 introduced new semantics. It encodes too much in-band and requires rewriting of the server data. A mitmproxy capture is higher fidelity and supports capturing modern features such as WebSockets. I think if we could wrap Kage's crawler interactions by it and store its capture (the intercepted traffic), we could make a…

I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support.

For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source), something that anyone can host, including the crawler, indexer, storage, and serving layers. Stay tuned!

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#18

Post by HelloUsername »

maxloh wrote: post #2

I find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli

What's the difference with, any webbrowser on a computer, File -> Save as ?

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#19

Post by dimiprasakis »

Neat project, I like the idea. One thing from a quick read: you launch Chrome with --no-sandbox. Is there a good reason for that? Security wise it's probably not a good idea. If there is no reason, I'd suggest leaving the sandbox on!

In any case, cool stuff :)

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#20

Post by nmstoker »

HelloUsername wrote: post #18
maxloh wrote: post #2

I find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli

What's the difference with, any webbrowser on a computer, File -> Save as ?

That's for a single page, this handles the whole site. Also the browser Save As options often work poorly.
Post reply on HN