Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

81–90 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#81

Post by masklinn »

js4ever wrote: post #70
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

It takes 2 commandes to create a new LXC container and ssh into it, then I install the software inside the container

That's more secure than running straight on the machine but not completely secure, security was never a core goal of linux containers.

You want to run the software properly sandboxed, since linux doesn't have really engineered OS-level solutions (à la Solaris) that means vm e.g. running in qemu, or biting the bullet and switching to qubes.

Re: Show HN: CLI tool for saving web pages as a single file

#82

Post by chrisked »

FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

I just clicked on the OP user name first which has the caption Bad boy. Then I read your comment. Now I worry more about the security aspect than before.

Re: Show HN: CLI tool for saving web pages as a single file

#83

Post by lucasverra »

super project ! i ve pretty baffled with the difficulty to save a webpage in proper format. I’ve tried with PDF converter, getPolaroid app and of course firefox screen shot feature for the entire scroll thing. Will try this for saving purposes.

I am also interested in cloning/forking sites for modification purposes, I will feedback you on the results four my consulting gigs

Re: Show HN: CLI tool for saving web pages as a single file

#84

Post by emerongi »

masklinn wrote: post #41
tenken wrote: post #35

How is this different from https://en.m.wikipedia.org/wiki/Web_ARChive

It looks like it creates a normal HTML file (embedding assets as data URI) so it should require no special client / support. HTMLD, WARC, MTHML, MAFF and webarchive are all "container" formats which bundle assets next to the HTML using various methods (resp. bundle, custom, multipart MIME, zip and binary plist).

The issue with this is that if the website requires some external API for content, it might not work properly.

https://webrecorder.io/ solves that problem by recording all interactions and then replaying them as needed.

> Webrecorder takes a new approach to web archiving by “recording” network traffic and processes within the browser while the user interacts with a web page. Unlike conventional crawl-based web archiving methods, this allows even intricate websites, such as those with embedded media, complex Javascript, user-specific content and interactions, and other dynamic elements, to be captured and faithfully restaged.

Re: Show HN: CLI tool for saving web pages as a single file

#85

Post by Huggernaut »

masklinn wrote: post #81
js4ever wrote: post #70

Earlier quoted context omitted.

It takes 2 commandes to create a new LXC container and ssh into it, then I install the software inside the container

That's more secure than running straight on the machine but not completely secure, security was never a core goal of linux containers. You want to run the software properly sandboxed, since linux doesn't have really engineered OS-level solutions (à la Solaris) that means vm e.g. running in qemu, or biting the bullet and switching to qubes.

I think that's an extremely uncharitable view on containers. There has been a massive amount of work put into securing containers for a variety of use cases using both layers available and by adding to the kernel.

Re: Show HN: CLI tool for saving web pages as a single file

#86

Post by fit2rule »

I've been printing to PDF for decades now, and nothing comes close to the ease of use and versatility of 2 decades worth of interesting web pages .. I have pretty much every interesting article, including many from HN, from decades of this habit.

Need to find all articles relating to 'widget'?

    $ ls -l ~/PDFArchive/ | grep -i widget
This has proven so valuable, time and again .. there is a great joy in not having to maintain bookmarks, and in being able to copy the whole directory content to other machines for processing/reference .. and then there's the whole pdf->text situation, which has its thorns truly (some website content is buried in masses of ad-noise), but also has huge advantage - there's a lot of data to be mined from 50,000 PDF files ..

Therefore, I'd quite like to know, what does monolith have to offer over this method? I can imagine that its useful to have all the scripting content packaged up and bundled into a single .html file - but does it still work/run? (This can be either a pro or a con in my opinion..)

Re: Show HN: CLI tool for saving web pages as a single file

#87

Post by ErrantX »

chrisked wrote: post #82
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

I just clicked on the OP user name first which has the caption Bad boy . Then I read your comment. Now I worry more about the security aspect than before.

Its funny how we sometimes think about risk; if the caption was 'good egg' would you be more trusting?

Re: Show HN: CLI tool for saving web pages as a single file

#88

Post by gildas »

jplayer01 wrote: post #36

Ah, I've been thinking about making something like this. You beat me to it. I've been using the SingleFile add-on until now. I'll definitely give this a try.

FYI, SingleFile can run on command line too, cf. https://github.com/gildas-lormeau/SingleFile/tree/master/cli

Re: Show HN: CLI tool for saving web pages as a single file

#90

Post by flatroze »

mikekchar wrote: post #58

With respect to the Unlicense, does anybody have any knowledge about how good it is in countries which don't allow you to intentionally pass things into the public domain (most countries that aren't the US)? How does it compare to CC0 in that respect?

Which license would you recommend to release this software under to reach the broad adoption yet permissive terms, if not the Unlicense?
Post reply on HN