Show HN: CLI tool for saving web pages as a single file
1–10 of 216 posts
Re: Show HN: CLI tool for saving web pages as a single file
#2This is awesome. One question though: how does it handle the same resource (e.g. image) appearing multiple times? Does it store multiple copies, potentially blowing up the file size? If not, how does it link to them in a single HTML file? If or if so, is there any way to get around it without using MHTML (or have you considered using MHTML in that case)?
Also, side-question about Rust: how do I get rid of absolute file paths in the executable to avoid information leakage? I feel like I partially figured this out at some point, but I forget.
Re: Show HN: CLI tool for saving web pages as a single file
#3Very cool.
Have you considered incorporating an option for following links within the same domain to a certain depth? I remember using tools such as this in the past to save all the content from certain websites.
Re: Show HN: CLI tool for saving web pages as a single file
#4Very cool. Have you considered incorporating an option for following links within the same domain to a certain depth? I remember using tools such as this in the past to save all the content from certain websites.
Thank you! I'll add it as an issue, since it could definitely be useful for "archiving" certain resources more than 1 level deep. Do you remember the name of that tool by any chance?
Re: Show HN: CLI tool for saving web pages as a single file
#5Very cool. Have you considered incorporating an option for following links within the same domain to a certain depth? I remember using tools such as this in the past to save all the content from certain websites.
Thank you! I'll add it as an issue, since it could definitely be useful for "archiving" certain resources more than 1 level deep. Do you remember the name of that tool by any chance?
I have used HTTrack in the past for this.
Re: Show HN: CLI tool for saving web pages as a single file
#6This would be a perfect fit for IPFS. I love the idea of having just one file in a permanent link.
Re: Show HN: CLI tool for saving web pages as a single file
#7This is awesome. One question though: how does it handle the same resource (e.g. image) appearing multiple times? Does it store multiple copies, potentially blowing up the file size? If not, how does it link to them in a single HTML file? If or if so, is there any way to get around it without using MHTML (or have you considered using MHTML in that case)? Also, side-question about Rust: how do I get rid of absolute fi…
https://github.com/Y2Z/monolith/blob/master/src/html.rs#L94
Hope this answers your question (it gets converted to a data URI, and there's apparently no de-duplication).
Re: Show HN: CLI tool for saving web pages as a single file
#8This is awesome. One question though: how does it handle the same resource (e.g. image) appearing multiple times? Does it store multiple copies, potentially blowing up the file size? If not, how does it link to them in a single HTML file? If or if so, is there any way to get around it without using MHTML (or have you considered using MHTML in that case)? Also, side-question about Rust: how do I get rid of absolute fi…
Thank you! It's pretty straight-forward: this program just retrieves assets and converts them into data-URLs (data:...), then replaces the original href/src attribute value, so in case with the same image being linked multiple times, monolith will for sure bloat the output with the same base64 data, correct. I haven't looked into MHTMTL, ashamed to admit it's the first time I'm hearing about that format. I need to do some research, maybe I could improve monolith to overcome issues related to file size, thank you for the tip!
And about Rust: I think you're way ahead of me here as well, this is my first Rust program. If you're talking about it embedding some debug info into the binary which may include things like /home/dataflow then perhaps there's a compiler option for cargo or a way to strip the binary after it's compiled. ¯\_(ツ)_/¯ Sorry, that's the best I can tell at the moment.
Re: Show HN: CLI tool for saving web pages as a single file
#9If the output were a tar file, couldn’t we also say it was saving web pages as a single file? Wouldn’t that also be easier?
Re: Show HN: CLI tool for saving web pages as a single file
#10MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).