Show HN: CLI tool for saving web pages as a single file

Post by AnIdiotOnTheNet » Fri, Aug 23, 2019, 3:27 PM UTC

forgotmypw wrote: post #146

OJFord wrote: post #129

Earlier quoted context omitted.

I used to use a VM (motivation being to run Linux on my locked-down corporate-imaged Macbook) and it was usable, but not as fast it could be if it had free reign over all CPU/RAM. But, at least the way I was doing it, it's not adding any security as discussed here, since you're doing everything in the VM so anything in the VM has access to everything just as if everything on the host anyway.

Scripts I run have access to my development environment for a free software project I already keep in a public repo. What they don't have access to is my actual desktop, nor yesterday's snapshot of the VM desktop. It's obviously not as fast as running GNU on bare metal, but it's fast enough for text work.

Post by jolmg » Fri, Aug 23, 2019, 3:33 PM UTC

mike-cardwell wrote: post #103

FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

These are the install instructions the docs say to use: $ git clone https://github.com/Y2Z/monolith.git $ cd monolith $ cargo install These are the ones I used: $ git clone https://github.com/Y2Z/monolith.git $ cd monolith $ sudo docker run --rm -w "$(pwd)" -v "$(pwd):$(pwd)" -u "$(id -u):$(id -g)" rust cargo install That isolated the build process. Similar method to isolate the execution of the built project: $ cd t…

Post by baby » Fri, Aug 23, 2019, 3:41 PM UTC

lucideer wrote: post #122

mikaelmorvan wrote: post #94

The main problem with your code is that you only handle simple web1 site. What about javascript execution ? If you replay your capture, you have no idea of what you will see on general Web2 website. The only way I know to capture a web page properly is to "execute" it on a browser. Gildas, the guy behind SingleFile ( https://github.com/gildas-lormeau/SingleFile ) is well aware of that and his approach realy works eve…

The capture includes JS, so this should work for most JS-dependent sites, with the exception of scripts loading other additional assets. Tbh, often those are superfluous, or egregious examples of bad web dev, so it seems a reasonable solution for most cases. SingleFile is a different approach, but it's a lot more involved/less convenient than a cli, and loading in something like WebDriver on the cli for this would be…

Post by mrieck » Fri, Aug 23, 2019, 3:42 PM UTC

Post by penagwin » Fri, Aug 23, 2019, 3:50 PM UTC

quickthrower2 wrote: post #68

alpb wrote: post #11

I think it would be way better to explain in the repository: - how do you handle images? - does it handle embedded videos? - does it handle JS? to what extent? - does it handle lazily loaded assets (i.e. images that load only when you scroll down, or JS that loads 3 seconds later after the page is loaded) In general, how does this work? The current readme doesn't do a decent job explaining what the tool exactly is. F…

It can’t handle JS completely because we can’t predict a programs behaviour using static analysis. See Halting Problem for example.