Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kage – Shadow any website to a single binary for offline viewing

github.com

Post reply on HN
Print view View original

61–70 of 152 posts

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#62

Post by ares623 »

gregwebs wrote: post #5

This seems like it has potential to create a lot of load on a site- are there settings to set how fast it clones or avoid images/videos? Is there a way to only get a subset of a website?

Just pretend you're an AI crawler problem solved

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#63

Post by throwaway219450 »

telesilla wrote: post #30

I've been using httrack ( https://www.httrack.com ) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.

Specifically for wikis, is there a reason you wouldn't use Kiwix? For non "official" releases it's more complicated, but there are some services to generate the ZIM files. The desktop reader app is pretty good in my experience.

https://wiki.openzim.org/wiki/Build_your_ZIM_file

EDIT: https://get.kiwix.org/en/solutions/applications/kiwix-reader...

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#67

Post by mgiampapa »

tamnd wrote: post #14
wolttam wrote: post #10

One use I'd have for this is company wikis that you want to give folks easy offline access to (maybe the wiki has documentation that's useful at sites that don't have cellular coverage). Cool! It would be especially cool to have a version that didn't require the separate serving process - even though it's nifty you can package up a whole site as a single binary. Maybe a single HTML entrypoint shim with a bit of javas…

Submitting this to Hacker News is the right place! Thanks for your idea. I will consider implementing that :) Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.

I think the zim flow was perfect for offline use. I know I will be making use of it as soon as I can figure out how to pass chrome the cookies so I can be signed into the site. Didn't see it in the page, but I didn't look closely yet.

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#68

Post by threecheese »

tamnd wrote: post #17
sanqui wrote: post #15

Earlier quoted context omitted.

That's neat! In my opinion, the WARC format is quite tricky and underspecified especially since HTTP2 introduced new semantics. It encodes too much in-band and requires rewriting of the server data. A mitmproxy capture is higher fidelity and supports capturing modern features such as WebSockets. I think if we could wrap Kage's crawler interactions by it and store its capture (the intercepted traffic), we could make a…

I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support. For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source)…

OK, sounds fascinating; following! (your GH)

Re: Show HN: Kage – Shadow any website to a single binary for offline viewing

#69

Post by stavros »

simonw wrote: post #38

I was intrigued to see how the demo GIF in the README was generated: https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it: ascii-gif render docs/demo/kage.tape -o docs/static/dem…

VHS is fantastic for scripting cli video generation.
Post reply on HN