Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

111–120 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#111

Post by gsich »

jasonvorhe wrote: post #80
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

What computer are you using and which operating system is running on that? Have you read the code?

Reading won't take you very far.

Re: Show HN: CLI tool for saving web pages as a single file

#112

Post by Springtime »

bhl wrote: post #75
Springtime wrote: post #10

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

One issue with MHTML is that it does not seem to be currently supported by iframes. The use case I was working on was comparing search results from Google and DuckDuckGo by simply scraping and downloading to later embed. For that, I used a cli tool from an open source library [1]. MHTML seems like a nice format but I'm not sure if there's a library to convert them into stand-alone HTML files. [1] https://github.com/g…

> MHTML seems like a nice format but I'm not sure if there's a library to convert them into stand-alone HTML files.

Many years ago used a program that could convert them but the name escapes me. A brief search shows a few results that appear to do a similar conversion, potentially they may be of use.

> This question just came to mind. If MHTML saves images using base64, and base64 dataurl images have a limit size, how would you save extremely large photos?

From what I've read there's no official limit for base64 encodings though IE/Edge limit them to 4GB according to caniuse.com. Haven't personally encountered an image or GIF that was too large not to be saved in an MHT (I have over 10k MHTML files saved, some image and GIF heavy ones up to 200MB each).

Also a sibling comment corrected me about the data URI use, MHTML uses a separate scheme but nevertheless still uses base64 for encoding.

For that example article you linked it seems likely to be the way the program you're using is handling the Javascript-loaded images. I saved it in Vivaldi (Blink-based engine) and the main image displayed at full res when opened locally while the other images didn't, while when saved with a pre-Quantum Firefox using the UnMHT addon it saved all the images at their fully loaded resolutions. Some MHTML saving implementations clearly have advantages over others it would seem.

Re: Show HN: CLI tool for saving web pages as a single file

#113

Post by dspillett »

flatroze wrote: post #8
mehrdadn wrote: post #2

This is awesome. One question though: how does it handle the same resource (e.g. image) appearing multiple times? Does it store multiple copies, potentially blowing up the file size? If not, how does it link to them in a single HTML file? If or if so, is there any way to get around it without using MHTML (or have you considered using MHTML in that case)? Also, side-question about Rust: how do I get rid of absolute fi…

Thank you! It's pretty straight-forward: this program just retrieves assets and converts them into data-URLs (data:...), then replaces the original href/src attribute value, so in case with the same image being linked multiple times, monolith will for sure bloat the output with the same base64 data, correct. I haven't looked into MHTMTL, ashamed to admit it's the first time I'm hearing about that format. I need to do…

One way to dedupe inline image resources while still using HTML rather than MHTML, could be to encode them in css once, and transform the image element to something with that class.

Re: Show HN: CLI tool for saving web pages as a single file

#115

Post by mjlee »

jasonvorhe wrote: post #80
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

What computer are you using and which operating system is running on that? Have you read the code?

I think you might get a kick out of Ken Thompson's Reflections on Trusting Trust.

https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...

Re: Show HN: CLI tool for saving web pages as a single file

#116

Post by dredmorbius »

fit2rule wrote: post #86

I've been printing to PDF for decades now, and nothing comes close to the ease of use and versatility of 2 decades worth of interesting web pages .. I have pretty much every interesting article, including many from HN, from decades of this habit. Need to find all articles relating to 'widget'? $ ls -l ~/PDFArchive/ | grep -i widget This has proven so valuable, time and again .. there is a great joy in not having to m…

Having gone this route in part myself, advantages of HTML or other more-structured file formats, if there is appropriate metadata markup:

- Allow for recording source and author information (PDF ... doesn't always provide this).

- Allows for full-text search.

- Allows for editing out annoyances.

I'll frequently go from HTML to some simplified representation (e.g., Markdown), and then re-generate formats that are useful elsewhere: HTML, PDF, ePub, etc.

Dumping from HTML to Markdown frequently makes cruft-removal far simpler, and the principle content of most pages is text. In rare instances, images are useful, and even more rarely, any multimedia content (video, audio, programmatic content).

What's depressing is the number of sites which screw with even basic HTML. E.g., the NY Times rarely use HTML tables for tabular representation, and instead use a homebrew combination of custom markup, CSS, and JS to much the same effect. Pretty, in situ, but brittle and transports exceedingly poorly.

That's just one of many such cases.

Re: Show HN: CLI tool for saving web pages as a single file

#118

Post by [deleted] »

mikaelmorvan wrote: post #94

The main problem with your code is that you only handle simple web1 site. What about javascript execution ? If you replay your capture, you have no idea of what you will see on general Web2 website. The only way I know to capture a web page properly is to "execute" it on a browser. Gildas, the guy behind SingleFile ( https://github.com/gildas-lormeau/SingleFile ) is well aware of that and his approach realy works eve…

[deleted]

Re: Show HN: CLI tool for saving web pages as a single file

#119

Post by pbhjpbhj »

masklinn wrote: post #106
Huggernaut wrote: post #85

Earlier quoted context omitted.

I think that's an extremely uncharitable view on containers. There has been a massive amount of work put into securing containers for a variety of use cases using both layers available and by adding to the kernel.

> I think that's an extremely uncharitable view on containers. It's an objective one. > There has been a massive amount of work put into securing containers for a variety of use cases using both layers available and by adding to the kernel. That doesn't change the fact that security was never the primary goal for containers, so secure containers were and are a bunch of tricks, kludges and prayers being built up in th…

Of course it says safer, if someone tells you something is entirely safe you know they're lying.

Re: Show HN: CLI tool for saving web pages as a single file

#120

Post by skinnymuch »

dredmorbius wrote: post #117
Exuma wrote: post #19

Saving this for later

FYI: "favorite" is one way of doing that through HN. Bookmarks, or downloads, externally.

Favorites is limited to a certain amount on HN before you start losing the oldest favorite.
Post reply on HN