Show HN: CLI tool for saving web pages as a single file • Hacker News Board

Re: Show HN: CLI tool for saving web pages as a single file

#21

Post by flatroze » Fri, Aug 23, 2019, 5:34 AM UTC

VvR-Ox wrote: post #17

Very cool idea - thank you for this! On question: How does it handle those cookie pop-ups, gdpr-warnings etc?

Oh, thank you kindly.

That's an interesting question. I think it depends on how the given modal is implemented, but closing them should technically work (unless the page is saved with JavaScript code removed [-j flag]). Those notifications can easily be removed from the saved file using any text editor, should be pretty easy if you know how to edit HTML code. I don't think removing it would violate anything since "this website" will no longer really be a website but rather a local document at that point.

Re: Show HN: CLI tool for saving web pages as a single file

#22

Post by dvcrn » Fri, Aug 23, 2019, 5:34 AM UTC

I'm not so experienced but how does this compare to .webarchive?

Re: Show HN: CLI tool for saving web pages as a single file

#23

Post by Causality1 » Fri, Aug 23, 2019, 5:36 AM UTC

Springtime wrote: post #10

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

Always struck me as quite odd MHTML fell out of favor. Back in the day when I wanted to preserve a web page it was the logical choice since you could just click "save as archive".

Re: Show HN: CLI tool for saving web pages as a single file

#24

Post by flatroze » Fri, Aug 23, 2019, 5:43 AM UTC

dvcrn wrote: post #22

I'm not so experienced but how does this compare to .webarchive?

The idea is almost identical, yet saving as .webarchive is only supported by Safari, and it's also not a plaintext format, hence can't be edited as easily.

Re: Show HN: CLI tool for saving web pages as a single file

#25

Post by flatroze » Fri, Aug 23, 2019, 5:43 AM UTC

Causality1 wrote: post #23

Springtime wrote: post #10

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

Always struck me as quite odd MHTML fell out of favor. Back in the day when I wanted to preserve a web page it was the logical choice since you could just click "save as archive".

We'll bring it back, don't you worry!

Re: Show HN: CLI tool for saving web pages as a single file

#26

Post by flatroze » Fri, Aug 23, 2019, 5:44 AM UTC

tetrisgm wrote: post #6

This would be a perfect fit for IPFS. I love the idea of having just one file in a permanent link.

This could also be an interesting alternative to PDF, especially with web fonts embedded as data URLs.

Re: Show HN: CLI tool for saving web pages as a single file

#27

Post by sbmthakur » Fri, Aug 23, 2019, 5:48 AM UTC

Nice work! I am wondering if Puppeteer can also be used to accomplish the same thing.

Re: Show HN: CLI tool for saving web pages as a single file

#28

Post by flatroze » Fri, Aug 23, 2019, 5:48 AM UTC

ajxs wrote: post #14

flatroze wrote: post #4

Earlier quoted context omitted.

Thank you! I'll add it as an issue, since it could definitely be useful for "archiving" certain resources more than 1 level deep. Do you remember the name of that tool by any chance?

I wish I could remember the exact tool, this was over a decade ago. If you just do a quick internet search for such a tool you'll likely find whatever I used, it certainly wasn't anything sophisticated. It was a Windows GUI tool designed specifically for the task. Something makes me think that 'GetRight' tool might have been able to do the same thing, but I can't seem to see the feature on their website.

Ah, I remember using something like that. I thought that tool was saving it into one .html file, but data URLs didn't exist back then, so creating directories alongside with HTML files was the only option to "replicate" a web resource, now I understand exactly what you were talking about. I'll do some more digging around and implement that in the nearest future. I may need to make all the requests async first to make sure that saving one resource with decent depth won't take too long.

Re: Show HN: CLI tool for saving web pages as a single file

#29

Post by flatroze » Fri, Aug 23, 2019, 6:06 AM UTC

sbmthakur wrote: post #27

Nice work! I am wondering if Puppeteer can also be used to accomplish the same thing.

It for sure would help with those SPA websites that get their DOM fully generated by JS. A web extension that saves the current DOM tree as HTML would perhaps do a better job, especially when it comes to resources which require some web-based authentication.

Re: Show HN: CLI tool for saving web pages as a single file

#30

Post by da_chicken » Fri, Aug 23, 2019, 6:13 AM UTC

Causality1 wrote: post #23

Springtime wrote: post #10

MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).

Always struck me as quite odd MHTML fell out of favor. Back in the day when I wanted to preserve a web page it was the logical choice since you could just click "save as archive".

Blame XMLHttpRequest, Flash, JS, and embedded video. It doesn't make sense to archive a document when the necessary interactive content elements will essentially fail when opened offline.