Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

11–20 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#11

Post by alpb »

I think it would be way better to explain in the repository:

- how do you handle images?

- does it handle embedded videos?

- does it handle JS? to what extent?

- does it handle lazily loaded assets (i.e. images that load only when you scroll down, or JS that loads 3 seconds later after the page is loaded)

In general, how does this work? The current readme doesn't do a decent job explaining what the tool exactly is. For all I can tell, it probably just takes a screenshot of the page, encodes as base64 into the html and shows it.

Re: Show HN: CLI tool for saving web pages as a single file

#12

Post by flatroze »

Springtime wrote: post #10

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

Apparently everybody knew about MHTML but me Ü I'm going to look into that format and see if I could enhance monolith to output proper MHTML, among other additions and improvements. Thank you for the info!

Re: Show HN: CLI tool for saving web pages as a single file

#13

Post by mehrdadn »

flatroze wrote: post #8
mehrdadn wrote: post #2

This is awesome. One question though: how does it handle the same resource (e.g. image) appearing multiple times? Does it store multiple copies, potentially blowing up the file size? If not, how does it link to them in a single HTML file? If or if so, is there any way to get around it without using MHTML (or have you considered using MHTML in that case)? Also, side-question about Rust: how do I get rid of absolute fi…

Thank you! It's pretty straight-forward: this program just retrieves assets and converts them into data-URLs (data:...), then replaces the original href/src attribute value, so in case with the same image being linked multiple times, monolith will for sure bloat the output with the same base64 data, correct. I haven't looked into MHTMTL, ashamed to admit it's the first time I'm hearing about that format. I need to do…

Okay thanks! That was a pretty quick reply :) Regarding MHTML, it's basically the MIME format emails are in (which are basically inherently self-contained HTML documents). Various browsers have had varying degrees of support for it over the years. Chrome recently made it harder to save in MHTML format; I don't know how long they will be able to read the format, so I can't guarantee that if you go in that direction it'll still be useful for a long time, but at the moment there is still some support for it.

Re: Show HN: CLI tool for saving web pages as a single file

#14

Post by ajxs »

flatroze wrote: post #4
ajxs wrote: post #3

Very cool. Have you considered incorporating an option for following links within the same domain to a certain depth? I remember using tools such as this in the past to save all the content from certain websites.

Thank you! I'll add it as an issue, since it could definitely be useful for "archiving" certain resources more than 1 level deep. Do you remember the name of that tool by any chance?

I wish I could remember the exact tool, this was over a decade ago. If you just do a quick internet search for such a tool you'll likely find whatever I used, it certainly wasn't anything sophisticated. It was a Windows GUI tool designed specifically for the task. Something makes me think that 'GetRight' tool might have been able to do the same thing, but I can't seem to see the feature on their website.

Re: Show HN: CLI tool for saving web pages as a single file

#15

Post by flatroze »

alpb wrote: post #11

I think it would be way better to explain in the repository: - how do you handle images? - does it handle embedded videos? - does it handle JS? to what extent? - does it handle lazily loaded assets (i.e. images that load only when you scroll down, or JS that loads 3 seconds later after the page is loaded) In general, how does this work? The current readme doesn't do a decent job explaining what the tool exactly is. F…

Good points, thank you for the review. I'll work on enhancing the readme file to be more informative.

Re: Show HN: CLI tool for saving web pages as a single file

#16

Post by darrenf »

flatroze wrote: post #4
ajxs wrote: post #3

Very cool. Have you considered incorporating an option for following links within the same domain to a certain depth? I remember using tools such as this in the past to save all the content from certain websites.

Thank you! I'll add it as an issue, since it could definitely be useful for "archiving" certain resources more than 1 level deep. Do you remember the name of that tool by any chance?

GNU wget supports recursive downloads: https://www.gnu.org/software/wget/manual/html_node/Recursive...

Re: Show HN: CLI tool for saving web pages as a single file

#20

Post by flatroze »

dfee wrote: post #9

If the output were a tar file, couldn’t we also say it was saving web pages as a single file? Wouldn’t that also be easier?

I think there's an issue with opening a tar file, e.g. if sent to someone who needs to view the document but isn't techy.

It seems to me that having one file that any browser can easily open (and not require Internet connection to view) is a big advantage over having a directory with assets alongside the .html file. It may be one of those things that make things easier yet nobody really complains about how things are usually done when the page gets saved. I hope more browsers add support for saving pages as MHTML in the nearest future so that we wouldn't need tools like this one.

Post reply on HN