Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

41–50 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#41

Post by masklinn »

tenken wrote: post #35

How is this different from https://en.m.wikipedia.org/wiki/Web_ARChive

It looks like it creates a normal HTML file (embedding assets as data URI) so it should require no special client / support.

HTMLD, WARC, MTHML, MAFF and webarchive are all "container" formats which bundle assets next to the HTML using various methods (resp. bundle, custom, multipart MIME, zip and binary plist).

Re: Show HN: CLI tool for saving web pages as a single file

#43

Post by sergioisidoro »

This sounds great, but the first thing I thought was how this would be a perfect tool to make automated mass phishing scams.

If the outcomes are realistic, take a massive list of sites, make a snapshot of each page, replace the POST login URLs with the phishers, deploy these individual HTML files, and spread the links through email.

I wonder how does this project handle forms.

Re: Show HN: CLI tool for saving web pages as a single file

#44

Post by flatroze »

fouc wrote: post #31

Sweet idea! I would especially like to be able to capture videos and pictures too. I suspect for saving videos, a good approach would be some sort of proxy + headless browser combination, where the proxy is responsible for saving a copy of all data the browser requests for. Thoughts?

Thanks! Pictures should work, I'll check more tags first thing tomorrow when I start working on improving it.

I use youtube-dl for youtube and other popular web services myself. Embedding a video source as a data URL could in theory work, but it'd be quite a long base64 line. Also, editing .html files with tens or hundreds of megabytes of base64 in them would perhaps be less than convenient.

Re: Show HN: CLI tool for saving web pages as a single file

#45

Post by jordwalke »

Springtime wrote: post #10

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

I'm not aware of a way to save as MHTML from Chrome in headless mode (from the command line). Are you?

Re: Show HN: CLI tool for saving web pages as a single file

#46

Post by jordwalke »

da_chicken wrote: post #30
Causality1 wrote: post #23

Earlier quoted context omitted.

Always struck me as quite odd MHTML fell out of favor. Back in the day when I wanted to preserve a web page it was the logical choice since you could just click "save as archive".

Blame XMLHttpRequest, Flash, JS, and embedded video. It doesn't make sense to archive a document when the necessary interactive content elements will essentially fail when opened offline.

You can first prerender the page with Chrome in headless mode (see my other comment), and then convert it into a single document using an inlining tool (such as the OP's). That way the JS will run and render the page (see my other comments here for an example).

Re: Show HN: CLI tool for saving web pages as a single file

#47

Post by flatroze »

jp_sc wrote: post #18
ajxs wrote: post #3

Very cool. Have you considered incorporating an option for following links within the same domain to a certain depth? I remember using tools such as this in the past to save all the content from certain websites.

SiteSucker for macOS does it.

Well, at least it's not called iSuck.

Re: Show HN: CLI tool for saving web pages as a single file

#48

Post by flatroze »

sergioisidoro wrote: post #43

This sounds great, but the first thing I thought was how this would be a perfect tool to make automated mass phishing scams. If the outcomes are realistic, take a massive list of sites, make a snapshot of each page, replace the POST login URLs with the phishers, deploy these individual HTML files, and spread the links through email. I wonder how does this project handle forms.

Thank you for reminding me, I need to set action="" to be an absolute path when the page is saved.

upd: Done, now forms get their action="/submit" converted into action="https://website.com/submit" when the page is saved.

Re: Show HN: CLI tool for saving web pages as a single file

#49

Post by rmetzler »

darrenf wrote: post #16
flatroze wrote: post #4

Earlier quoted context omitted.

Thank you! I'll add it as an issue, since it could definitely be useful for "archiving" certain resources more than 1 level deep. Do you remember the name of that tool by any chance?

GNU wget supports recursive downloads: https://www.gnu.org/software/wget/manual/html_node/Recursive...

IIRC this was the mode Snowden used to bulk-download the NSA data.
Post reply on HN