Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

71–80 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#71

Post by kuzehanka »

quickthrower2 wrote: post #68
alpb wrote: post #11

I think it would be way better to explain in the repository: - how do you handle images? - does it handle embedded videos? - does it handle JS? to what extent? - does it handle lazily loaded assets (i.e. images that load only when you scroll down, or JS that loads 3 seconds later after the page is loaded) In general, how does this work? The current readme doesn't do a decent job explaining what the tool exactly is. F…

It can’t handle JS completely because we can’t predict a programs behaviour using static analysis. See Halting Problem for example.

I saw a tool that handles JS to a limited extent by capturing and replaying network requests to accommodate said JS. It records your session while you interact with a site, and is then able to replay everything it captured.

This tool was able to capture three.js applications and other interactive sites quite well.

Re: Show HN: CLI tool for saving web pages as a single file

#72

Post by laumars »

interfixus wrote: post #65
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

We may semi-trust our package and repo systems. This tool is readily available through AUR on my Arch machine, I see. Or we may go the whole hog and actually have a peek through the source.

AUR are packages that aren't in Arch's repo system. Granted tools like yaourt do make installing AUR packages nearly as easy as pacman but anyone can upload anything to AUR thus you are expected to vet the packages yourself (hence why tools like yaourt repeatedly prompt you to read the build scripts et al before running them).

Re: Show HN: CLI tool for saving web pages as a single file

#73

Post by bhl »

jordwalke wrote: post #45
Springtime wrote: post #10

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

I'm not aware of a way to save as MHTML from Chrome in headless mode (from the command line). Are you?

You can either use the flag --save-page-as-mhtml from CLI [1] or a library such as puppeteer.

[1] https://techdows.com/2019/06/google-removes-save-page-as-mht...

[2] https://github.com/GoogleChrome/puppeteer

Re: Show HN: CLI tool for saving web pages as a single file

#74

Post by tannhaeuser »

Well you could do that for a long time with MHTML, WARC, etc. downloaders, including those available in browsers via "Save Page as", though CSS imports aren't covered by older tools (are they by yours?). Anyway, congrats for completing this as a Rust first-timer project, which certainly speaks to the quality of the Rust ecosystem. For using this approach as offline browser, of course, the problem is that Ajax-heavy pages using Javascript for loading content won't work, including every React and Vue sites created in the last five years (but you could make the point those aren't worth your attention as a reader anyway).

Re: Show HN: CLI tool for saving web pages as a single file

#75

Post by bhl »

Springtime wrote: post #10

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

One issue with MHTML is that it does not seem to be currently supported by iframes. The use case I was working on was comparing search results from Google and DuckDuckGo by simply scraping and downloading to later embed. For that, I used a cli tool from an open source library [1]. MHTML seems like a nice format but I'm not sure if there's a library to convert them into stand-alone HTML files.

[1] https://github.com/gildas-lormeau/SingleFile

Edit: This question just came to mind. If MHTML saves images using base64, and base64 dataurl images have a limit size, how would you save extremely large photos? Take for example the cover image of this article https://story.californiasunday.com/gone-paradise-fire. When I saved the page in MHTML format, the re-rendered image showed up quite blurry compared to the original. Was the size limit the cause?

[2] https://stackoverflow.com/questions/12637395/what-is-the-siz...

Re: Show HN: CLI tool for saving web pages as a single file

#76

Post by donatzsky »

flatroze wrote: post #38
makach wrote: post #33

Ahh, to me it looks like it creates an amalgamation of the web page+contents. How does this work on neverending webpages/forever scroll? How will it behave if you need to authenticate before browsing the page?

That's it in the nutshell! It seems to work for basic pages quite well, I think that lazy load will work for most pages as long as the JavaScript is embedded (no -j flag provided) and the Internet connection is on. It saves what's there when the page is loaded, the rest is a gamble since every website implements infinite scroll differently. Authentication is another tricky part -- it's different for every browser. I…

For authentication, you could add an option for passing http headers, as well as accept Netscape-style cookie files.

Whenever I want to download a video, using YouTube-dl, from a site that requires authentication, I first login using my browser and then exports the cookies using an extension.

Re: Show HN: CLI tool for saving web pages as a single file

#77

Post by flatroze »

tannhaeuser wrote: post #74

Well you could do that for a long time with MHTML, WARC, etc. downloaders, including those available in browsers via "Save Page as", though CSS imports aren't covered by older tools (are they by yours?). Anyway, congrats for completing this as a Rust first-timer project, which certainly speaks to the quality of the Rust ecosystem. For using this approach as offline browser, of course, the problem is that Ajax-heavy p…

CSS imports are covered by converting .css files into data URLs, later I will parse those and embed resources found within stylesheets as well.

Re: Show HN: CLI tool for saving web pages as a single file

#78

Post by bhl »

kuzehanka wrote: post #71
quickthrower2 wrote: post #68

Earlier quoted context omitted.

It can’t handle JS completely because we can’t predict a programs behaviour using static analysis. See Halting Problem for example.

I saw a tool that handles JS to a limited extent by capturing and replaying network requests to accommodate said JS. It records your session while you interact with a site, and is then able to replay everything it captured. This tool was able to capture three.js applications and other interactive sites quite well.

Was it webrecorder [1]? I found this project a couple weeks back while looking for web archiving tools.

[1] https://webrecorder.io/

Re: Show HN: CLI tool for saving web pages as a single file

#79

Post by kuzehanka »

bhl wrote: post #78
kuzehanka wrote: post #71

Earlier quoted context omitted.

I saw a tool that handles JS to a limited extent by capturing and replaying network requests to accommodate said JS. It records your session while you interact with a site, and is then able to replay everything it captured. This tool was able to capture three.js applications and other interactive sites quite well.

Was it webrecorder [1]? I found this project a couple weeks back while looking for web archiving tools. [1] https://webrecorder.io/

Yep, that's the one! Thanks for reminding the name.

Re: Show HN: CLI tool for saving web pages as a single file

#80

Post by jasonvorhe »

FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

What computer are you using and which operating system is running on that? Have you read the code?
Post reply on HN