Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

181–190 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#181

Post by derefr »

gambler wrote: post #175
derefr wrote: post #165

Earlier quoted context omitted.

But arbitrary programs are... arbitrary. Especially ones run by software engineers, and especially ones run by software engineers as part of a POSIX-alike “utility bag” ecosystem. Who’s to say that the user’s intent by running the program they just downloaded, isn’t to—say—overwrite a system folder? (Oh, wait, that’s exactly what Homebrew does, with the user’s full intent behind it!) There are tons of attempts to do…

>But what if you’re attempting to use those utilities against your real documents? You copy or move documents inside the specific sandbox. If you want a pipeline, you establish a chain of inbox/outbox folders. Obviously, most of this should be done by the OS, not the user. The workflow: - You click "download" in your browser. - When it's done and you click on your download, the OS asks how you want to open it. Instea…

Let me put it another way: how would you implement a dotfile management framework (like any of these: https://dotfiles.github.io)? Programmers seem to really like them, judging by how many of them there are. But the whole point of them is to forcefully usurp the assets of literally every other program on the system. They're user-level rootkits, in a sense.

Or, for a simpler, more obvious example: find(1), grep(1), etc. A set of utilities that can all be asked the equivalent of "read literally every file the VFS has access to and tell me whether they match an arbitrary-code-execution predicate." Do you want to literally copy your entire hard disk into the 'inbox' of these utilities, in order to get them to search it for you? (And before you say "well, we can trust the base utilities that ship with the OS to do more than arbitrary third-party utilities"—there's a whole competition of grep(1) replacements, e.g. ag(1), rg(1), etc. Do you want to make it impossible for people to innovate in this space?)

Or how about Nix, or GNU Stow, or, uh, Git? These utilities become useless if they have their own sandbox. Does your git worktree live in Git's sandbox? Vi's sandbox? The inability to make this distinction functional is why mobile OSes only have fullly-integrated IDEs!

Or how about shells themselves! (Or, equivalently, any scripting runtime, e.g. Ruby, Python, etc.) Should people not be allowed to install these from third parties?

Or, the most based example of all: make(1) [and its spiritual descendants], and the GNU autotools built atop it. How does ./configure work if you can't detect true properties of the target system, only of the sandbox you're in?

Re: Show HN: CLI tool for saving web pages as a single file

#182

Post by gildas »

sametmax wrote: post #168

Good, but won't work with the heavy JS pages using Ajax to load any single content. The firefox extension seems to do that : https://addons.mozilla.org/fr/firefox/addon/single-file/

Unfortunately it's not written in Rust so it won't make the first page of HN.

Re: Show HN: CLI tool for saving web pages as a single file

#183

Post by Avi-D-coder »

FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

Before I installed this tool, I checked if the author was a member of any well known organization and since they are not, I skimmed all the code for monolith and any of it's dependency that are not extremely popular, in this case mime-sniffer.

Re: Show HN: CLI tool for saving web pages as a single file

#184

Post by mehrdadn »

dspillett wrote: post #113
flatroze wrote: post #8

Earlier quoted context omitted.

Thank you! It's pretty straight-forward: this program just retrieves assets and converts them into data-URLs (data:...), then replaces the original href/src attribute value, so in case with the same image being linked multiple times, monolith will for sure bloat the output with the same base64 data, correct. I haven't looked into MHTMTL, ashamed to admit it's the first time I'm hearing about that format. I need to do…

One way to dedupe inline image resources while still using HTML rather than MHTML, could be to encode them in css once, and transform the image element to something with that class.

That'd easily break Javascript though.

Re: Show HN: CLI tool for saving web pages as a single file

#185

Post by Hackbraten »

derefr wrote: post #165
gambler wrote: post #150

Earlier quoted context omitted.

This is a good question. I think you can make it even better by generalizing the problem. How on earth do developers hope to advance general computing forward when simply running programs isn't a solved problem? Most software engineers I know don't run docker on their home PCs. What about people who aren't in IT? Does anyone here even care? The general attitude I see is "plebs don't need to run anything they can't ge…

But arbitrary programs are... arbitrary. Especially ones run by software engineers, and especially ones run by software engineers as part of a POSIX-alike “utility bag” ecosystem. Who’s to say that the user’s intent by running the program they just downloaded, isn’t to—say—overwrite a system folder? (Oh, wait, that’s exactly what Homebrew does, with the user’s full intent behind it!) There are tons of attempts to do…

> Oh, wait, that’s exactly what Homebrew does, with the user’s full intent behind it!

Care to elaborate?

Re: Show HN: CLI tool for saving web pages as a single file

#186

Post by derefr »

Hackbraten wrote: post #185
derefr wrote: post #165

Earlier quoted context omitted.

But arbitrary programs are... arbitrary. Especially ones run by software engineers, and especially ones run by software engineers as part of a POSIX-alike “utility bag” ecosystem. Who’s to say that the user’s intent by running the program they just downloaded, isn’t to—say—overwrite a system folder? (Oh, wait, that’s exactly what Homebrew does, with the user’s full intent behind it!) There are tons of attempts to do…

> Oh, wait, that’s exactly what Homebrew does, with the user’s full intent behind it! Care to elaborate?

Homebrew is designed to take over your OS /usr/local directory. Not that there's much in there by default, but Homebrew's presence greatly changes the semantics of /usr/local, given that it's normally meant as a prefix for the local system administrator to install things as root into, whereas Homebrew re-assigns ownership of the whole directory structure to the user installing Homebrew (who, admittedly, is a member of the "staff" group, but still only one member on a potentially multiuser computer.)

Re: Show HN: CLI tool for saving web pages as a single file

#187

Post by gambler »

derefr wrote: post #181
gambler wrote: post #175

Earlier quoted context omitted.

>But what if you’re attempting to use those utilities against your real documents? You copy or move documents inside the specific sandbox. If you want a pipeline, you establish a chain of inbox/outbox folders. Obviously, most of this should be done by the OS, not the user. The workflow: - You click "download" in your browser. - When it's done and you click on your download, the OS asks how you want to open it. Instea…

Let me put it another way: how would you implement a dotfile management framework (like any of these: https://dotfiles.github.io )? Programmers seem to really like them, judging by how many of them there are. But the whole point of them is to forcefully usurp the assets of literally every other program on the system. They're user-level rootkits, in a sense. Or, for a simpler, more obvious example: find(1), grep(1), e…

>"Do you want to literally copy your entire hard disk into the 'inbox' of these utilities, in order to get them to search it for you?"

Well, let's think about the goal here. grep reads files and outputs lines from those files. It needs full read access to everything you want to search. It does not need write access outside of its sandbox. It does not need direct access to network sockets, audio stuff and so on.

Is it unreasonable to create a readonly "view" of the filesystem inside grep's folder? Is it unreasonable to have "files" representing network access, microphone, audio? It will have visual representation in file manager without the need to create custom UI. It could be manipulated by drag-and-drop OR command line. More importantly: it's easy for users to understand. "This app lives in a box. You can put things in that box for the app to use."

>Does your git worktree live in Git's sandbox?

Yes? I mean, I currently have a folder called projects. All my git stuff is in there anyway.

>Vi's sandbox?

If you want multiple sandboxes to be able to operate on a directory, you create "views" for that directory (readonly or read/write) in multiple sandboxes. This shouldn't be some sort of mind-bending idea, considering Unix has symlinks, hardlinks, and mounted filesystems of all sorts.

>Or how about shells themselves! (Or, equivalently, any scripting runtime, e.g. Ruby, Python, etc.) Should people not be allowed to install these from third parties?

There is no reason why a Ruby executable should have unlimited access to the entire file system. Especially if you're only using it for a specific purpose, like serving a website.

What I'm describing here isn't some novel, mind-blowing idea. It's simply dependency injection. With file-based user interface. Every single part of this had been done in various operating systems or programming environments more than once. It's just a matter of combining it all in a sensible way.

Re: Show HN: CLI tool for saving web pages as a single file

#188

Post by cr0sh »

This is interesting - I think any of us who save things off the internet have made something like this (I usually save entire sites or large chunks, though - so I have a different toolset - still, I also do single pages, so I might try out this tool).

One thing I would propose to add - either a flag, or by default - have it parse the path to the page and create the file with the name - that way you can just "monolith {url}" and not have to worry about it.

I am also curious as to how it handles advertisements and google tracking and such; some way to strip out just those scripts (and elements) could be handy.

Re: Show HN: CLI tool for saving web pages as a single file

#189

Post by ur-whale »

flatroze wrote: post #174
ur-whale wrote: post #149

Does not compile with some byzantine message about let in const funcs being unstable.

Could you please open issue on github providing the output that you get in the terminal?

I have closed my github account since the takeover occured.

Re: Show HN: CLI tool for saving web pages as a single file

#190

Post by OJFord »

forgotmypw wrote: post #146
OJFord wrote: post #129

Earlier quoted context omitted.

I used to use a VM (motivation being to run Linux on my locked-down corporate-imaged Macbook) and it was usable, but not as fast it could be if it had free reign over all CPU/RAM. But, at least the way I was doing it, it's not adding any security as discussed here, since you're doing everything in the VM so anything in the VM has access to everything just as if everything on the host anyway.

Scripts I run have access to my development environment for a free software project I already keep in a public repo. What they don't have access to is my actual desktop, nor yesterday's snapshot of the VM desktop. It's obviously not as fast as running GNU on bare metal, but it's fast enough for text work.

Ah okay, that's why I thought to include 'at least for me' - because our 'all my development's were different! :)

For me, the VM was my entire machine. (It wasn't meant to improve security, it was purely because I wanted Linux but couldn't have it on the host.)

Post reply on HN