Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kage – Shadow any website to a single binary for offline viewing

github.com

Post reply on HN
Print view View original

71–80 of 152 posts

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#71

Post by telesilla »

throwaway219450 wrote: post #63
telesilla wrote: post #30

I've been using httrack ( https://www.httrack.com ) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.

Specifically for wikis, is there a reason you wouldn't use Kiwix? For non "official" releases it's more complicated, but there are some services to generate the ZIM files. The desktop reader app is pretty good in my experience. https://wiki.openzim.org/wiki/Build_your_ZIM_file EDIT: https://get.kiwix.org/en/solutions/applications/kiwix-reader...

Thanks, never knew about this and great to hear about it.

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#72

Post by vqtska »

simonw wrote: post #38

I was intrigued to see how the demo GIF in the README was generated: https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it: ascii-gif render docs/demo/kage.tape -o docs/static/dem…

You can also do an animated svg which is way smaller than a gif because it's just text keyframes (https://github.com/vytskalt/pseudoc/blob/main/assets/factori...)

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#73

Post by nikisweeting »

dimiprasakis wrote: post #19

Neat project, I like the idea. One thing from a quick read: you launch Chrome with --no-sandbox. Is there a good reason for that? Security wise it's probably not a good idea. If there is no reason, I'd suggest leaving the sandbox on! In any case, cool stuff :)

--no-sandbox is needed in docker, maybe they assume it will mostly run in docker?

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#74

Post by nikisweeting »

telesilla wrote: post #30

I've been using httrack ( https://www.httrack.com ) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.

https://github.com/archiveteam/grab-site or browsertrix may be easier to use for some, it's what was used to save a lot of the data.gov stuff before it got taken down.

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#75

Post by tamnd »

simonw wrote: post #38

I was intrigued to see how the demo GIF in the README was generated: https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it: ascii-gif render docs/demo/kage.tape -o docs/static/dem…

I have a bunch of opinionated/personal-use binaries like this in my $HOME/bin/, like delete-all-npm, clean-rust-cache, download-youtube-playlist, and get-markdown . It feels good, and I don't need to remember any commands. Sometimes my coding agent can figure out how to call some of those tools too ;))

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#76

Post by tamnd »

ninalanyon wrote: post #36

> kage serve $HOME/data/kage/paulgraham.com If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like: $ firefox $HOME/data/kage/paulgraham.com Then the result would be useable on machines without kage nstalled.

You could use python -m http.server instead. I haven't tried it yet, but it should work.

Actually, Kage has two parts: a crawler that crawls pages and converts them to clean HTML by capturing the DOM after rendering in Chrome/Chromium, and a pack/serve component that packages the result as either a ZIM file for Kiwix or an executable file.

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#77

Post by tamnd »

telesilla wrote: post #30

I've been using httrack ( https://www.httrack.com ) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.

This brings back memories. Around twenty years ago, internet was still expensive dial-up, so I used to go to an internet cafe, run HTTrack to download websites and manga, copy everything onto my tiny 128MB USB stick (felt very large at that time), then bring it home and read offline ;))

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#78

Post by tamnd »

throwaway219450 wrote: post #63
telesilla wrote: post #30

I've been using httrack ( https://www.httrack.com ) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.

Specifically for wikis, is there a reason you wouldn't use Kiwix? For non "official" releases it's more complicated, but there are some services to generate the ZIM files. The desktop reader app is pretty good in my experience. https://wiki.openzim.org/wiki/Build_your_ZIM_file EDIT: https://get.kiwix.org/en/solutions/applications/kiwix-reader...

Kiwix has readers for almost every platform, Android, desktop, iPhone. That's why I made Kage produce ZIM file.

The executable file is mostly for people who don't have Kiwix installed yet, or just want to run the archive directly.

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#79

Post by tamnd »

threecheese wrote: post #68
tamnd wrote: post #17

Earlier quoted context omitted.

I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support. For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source)…

OK, sounds fascinating; following! (your GH)

Thanks ;)

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#80

Post by tamnd »

calrizien wrote: post #64

Does this work for the Apple Docs website? Really tricky to get those offline.

Making docs available offline was one of my main motivations for building this tool. I will try Apple Docs too.

I previously downloaded the Snowflake docs, and it was something like tens or even hundreds of thousands of pages, I do not remember exactly. The output ended up being very large.

By the way, I forgot to add zstd compression support to my ZIM reader/writer. I will implement that in the next version.

Post reply on HN