Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

51–60 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#51

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

Apparently everybody knew about MHTML but me Ü I'm going to look into that format and see if I could enhance monolith to output proper MHTML, among other additions and improvements. Thank you for the info!

I don't know that it would be a very useful thing to do at least in the short term: there's a bunch of "web archive" formats out there and the common thread between them is that they're custom archive formats, you need special clients or support for those formats:

* mthml encodes the page as a multipart MIME message (using multipart/related), essentially an email (you're usually able to open them by replacing the .mth by .eml)

* WARC is its own thing with its own spec

* WAFF is a zipfile, not sure about the specifics

* webarchive is a binary plist, not sure about the specifics either

Your tool generates straight HTML which any browser should be able to open. It probably has more limitations, but it doesn't require dedicated client / viewer support.

Maybe once you've got all the fetching and extracting and linking nailed down it would be a nice extension to add "output filters", but that seems more like a secondary long-term goal, especially as those archive formats are usually semi-proprietary and get dropped as fast as they get created (WARC might be the most long-lived as it descends from the Internet Archive's ARC, is an ISO standard and is recognised as a proper archival format by various national libraries).

Re: Show HN: CLI tool for saving web pages as a single file

#53

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

It appears that (at least) Safari cannot open mhtml files. The benefit of a tool such as what the OP shared is that it can produce plain html pages that are openable by anyone. (also, I tried mhtml in Chrome using the proper flag and it doesn't appear to store/inline/render static assets correctly).

Re: Show HN: CLI tool for saving web pages as a single file

#54

I really like this concept, and I've been using an npm package called inliner which does this too: https://www.npmjs.com/package/inliner I'm glad there's more people taking a look at the use case, and I'd be interested to see a list of similar solutions. If you combine this with Chrome's headless mode, you can prerender many pages that use JavaScript to perform the initial render, and then once you're done send it to…

Thanks, I'll look into making it work with pipes or some other way to interact with headless browsers.

Re: Show HN: CLI tool for saving web pages as a single file

#55

This sounds great, but the first thing I thought was how this would be a perfect tool to make automated mass phishing scams. If the outcomes are realistic, take a massive list of sites, make a snapshot of each page, replace the POST login URLs with the phishers, deploy these individual HTML files, and spread the links through email. I wonder how does this project handle forms.

I'd imagine you can already do that with some basic web scraping tools. This definitely makes it easier.

Re: Show HN: CLI tool for saving web pages as a single file

#57
post #11

I think it would be way better to explain in the repository: - how do you handle images? - does it handle embedded videos? - does it handle JS? to what extent? - does it handle lazily loaded assets (i.e. images that load only when you scroll down, or JS that loads 3 seconds later after the page is loaded) In general, how does this work? The current readme doesn't do a decent job explaining what the tool exactly is. F…

[deleted]
Post reply on HN