Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

101–110 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#101

Post by sieabahlpark »

flatroze wrote: post #67
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

It's a valid question. It seems to me users tend to trust things which have certain level of popularity and reputation associated with them. I personally prefer to hope for the worst. This way when nothing happens I feel extra lucky, and if bad things do happen, I feel proud of being ready for it.

That could be said about nearly everything that you do.

Re: Show HN: CLI tool for saving web pages as a single file

#102

Post by OJFord »

FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

Some people run ~everything in individual containers, with original commands being aliases for the docker (or whatever) equivalent.

I've shunned that assuming it'd be a big slow down, but I do keep meaning to at least try it, uh, after I knock it.

(No, containers like anything else aren't and haven't been completely secure all of the time since and forever, but it'd take a more sophisticated - and certainly deliberately malicious - tool to do any damage to your system, or to files you didn't explicitly allow it access to.)

Re: Show HN: CLI tool for saving web pages as a single file

#103

Post by mike-cardwell »

FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

These are the install instructions the docs say to use:

    $ git clone https://github.com/Y2Z/monolith.git
    $ cd monolith
    $ cargo install
These are the ones I used:

    $ git clone https://github.com/Y2Z/monolith.git
    $ cd monolith
    $ sudo docker run --rm -w "$(pwd)" -v "$(pwd):$(pwd)" -u "$(id -u):$(id -g)" rust cargo install
That isolated the build process. Similar method to isolate the execution of the built project:

    $ cd target/release
    $ sudo docker run --rm -w "$(pwd)" -v "$(pwd):$(pwd)" -u "$(id -u):$(id -g)" rust ./monolith https://www.grepular.com
Slightly related: I released a project last night for easily running node applications inside containers: https://gitlab.com/mikecardwell/safernode - Without even having node or npm installed on the host system, you can still run commands like "npm install" or "npm start" to run node applications safely isolated inside ephemeral containers.

Re: Show HN: CLI tool for saving web pages as a single file

#104

Post by flatroze »

lucasverra wrote: post #83

super project ! i ve pretty baffled with the difficulty to save a webpage in proper format. I’ve tried with PDF converter, getPolaroid app and of course firefox screen shot feature for the entire scroll thing. Will try this for saving purposes. I am also interested in cloning/forking sites for modification purposes, I will feedback you on the results four my consulting gigs

Thank you for the kind words.

It will evolve into a reliable tool in a couple weeks and it should eventually work for embedding everything, including things like web fonts and @url()'s within CSS. If anything doesn't work, please open an issue, I have plenty of time to work on it.

Re: Show HN: CLI tool for saving web pages as a single file

#105

Post by flatroze »

Deukhoofd wrote: post #62
personjerry wrote: post #56

`cargo install` install 237 packages for this?! I don't think that's acceptable.

Probably the reqwest crate. That thing alone uses like 30 crates, not including the dependencies of those crates.

The compile time is rather long as well, I'm looking into ways of reducing the amount of dependencies.

Re: Show HN: CLI tool for saving web pages as a single file

#106

Post by masklinn »

Huggernaut wrote: post #85
masklinn wrote: post #81

Earlier quoted context omitted.

That's more secure than running straight on the machine but not completely secure, security was never a core goal of linux containers. You want to run the software properly sandboxed, since linux doesn't have really engineered OS-level solutions (à la Solaris) that means vm e.g. running in qemu, or biting the bullet and switching to qubes.

I think that's an extremely uncharitable view on containers. There has been a massive amount of work put into securing containers for a variety of use cases using both layers available and by adding to the kernel.

> I think that's an extremely uncharitable view on containers.

It's an objective one.

> There has been a massive amount of work put into securing containers for a variety of use cases using both layers available and by adding to the kernel.

That doesn't change the fact that security was never the primary goal for containers, so secure containers were and are a bunch of tricks, kludges and prayers being built up in the hope that eventually all the holes in the model will be patched.

The "Making containers safer"[lwn][hn] talk was literally two days ago. Note how it says safer, not safe.

[lwn] https://lwn.net/SubscriberLink/796700/9bc9daa32a8fe499/

[hn] https://news.ycombinator.com/item?id=20764355

Re: Show HN: CLI tool for saving web pages as a single file

#107

Post by flatroze »

fit2rule wrote: post #86

I've been printing to PDF for decades now, and nothing comes close to the ease of use and versatility of 2 decades worth of interesting web pages .. I have pretty much every interesting article, including many from HN, from decades of this habit. Need to find all articles relating to 'widget'? $ ls -l ~/PDFArchive/ | grep -i widget This has proven so valuable, time and again .. there is a great joy in not having to m…

I'd say since monolith produces a plaintext document it lets you edit things easier if needed.

JS can be removed from the final document using the -j flag. HTML Files can also be grepped for content, unlike PDFs.

Re: Show HN: CLI tool for saving web pages as a single file

#108

Post by flatroze »

interfixus wrote: post #69

Nice. I can see some automated uses for this. In ordinary browsing, am currently using a Firefox addon called SingleFile which works surprisingly well. Stuffs everything into (surprise, surprise) one huge single file - html with embedded data, so compatible everywhere.

It sounds like a great add-on, I have to check it out to see what it does to remote assets and how it works with asynchronously loaded assets.

Re: Show HN: CLI tool for saving web pages as a single file

#109

Post by interfixus »

laumars wrote: post #72
interfixus wrote: post #65

Earlier quoted context omitted.

We may semi-trust our package and repo systems. This tool is readily available through AUR on my Arch machine, I see. Or we may go the whole hog and actually have a peek through the source.

AUR are packages that aren't in Arch's repo system. Granted tools like yaourt do make installing AUR packages nearly as easy as pacman but anyone can upload anything to AUR thus you are expected to vet the packages yourself (hence why tools like yaourt repeatedly prompt you to read the build scripts et al before running them).

> AUR are packages that aren't in Arch's repo system

AUR, Arch User Repository.

> thus you are expected to vet the packages yourself

Obviously, as a long time Arch user I didn't know this.

Re: Show HN: CLI tool for saving web pages as a single file

#110

Post by forgotmypw »

OJFord wrote: post #102
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

Some people run ~everything in individual containers, with original commands being aliases for the docker (or whatever) equivalent. I've shunned that assuming it'd be a big slow down, but I do keep meaning to at least try it, uh, after I knock it. (No, containers like anything else aren't and haven't been completely secure all of the time since and forever, but it'd take a more sophisticated - and certainly deliberat…

I do all of my development in a VM, which allows me to take snapshots and have a portable GNU workstation that's decoupled from my desktop and hardware.

Meanwhile, my desktop remains clean and ready to play media in native environment with good hardware support.

I used to think it'd be slow, until I tried it. My computer is 8+ years old, and it works fine. I mostly do text work.

Post reply on HN