Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

171–180 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#171

Post by vaer-k »

gambler wrote: post #150
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

This is a good question. I think you can make it even better by generalizing the problem. How on earth do developers hope to advance general computing forward when simply running programs isn't a solved problem? Most software engineers I know don't run docker on their home PCs. What about people who aren't in IT? Does anyone here even care? The general attitude I see is "plebs don't need to run anything they can't ge…

> I imagine something like "applications" folder where every subfolder automatically becomes an isolated "container". It would have to be designed with security as the primary concern, though; unlike current container solutions.

Isn't that basically just Qubes OS? https://www.qubes-os.org/

Re: Show HN: CLI tool for saving web pages as a single file

#172

Post by geggam »

mike-cardwell wrote: post #163
jolmg wrote: post #153

Earlier quoted context omitted.

Though I don't know the specifics, isn't it commonly advised to not rely on docker for secure isolation of potential malware?

I've heared this too, but as far as I know it's only because there are potential bugs in the container software that allow the malware to escape. To me, this is kind of like saying you should just run stuff as root, because there might be a privelege escalation vulnerability which lets the code run as root anyway. Correct me if I'm wrong. My goal was to make things more secure, not completely secure. Previously, dodg…

A full VM seems somewhat of a better stance on this no ?

Re: Show HN: CLI tool for saving web pages as a single file

#173

Post by cookiecaper »

jolmg wrote: post #164
mike-cardwell wrote: post #163

Earlier quoted context omitted.

I've heared this too, but as far as I know it's only because there are potential bugs in the container software that allow the malware to escape. To me, this is kind of like saying you should just run stuff as root, because there might be a privelege escalation vulnerability which lets the code run as root anyway. Correct me if I'm wrong. My goal was to make things more secure, not completely secure. Previously, dodg…

> To me, this is kind of like saying you should just run stuff as root, because there might be a privelege escalation vulnerability which lets the code run as root anyway. But I never said it was preferable to run directly on the host. There are other choices. > My goal was to make things more secure, not completely secure. There is no such thing as completely secure. The argument against docker is more along the lin…

Linux relies on a concoction of properly-configured kernel subsystems to provide some level of isolation for containerized processes, and systems like LXD and Docker try to patch up the gaps.

The cgroups interfaces don't offer much security stuff directly -- they're mainly about containing groups of process within certain resource consumption quotas, and afaik, don't really attempt to contemplate secure isolation directly.

LXD approaches this by adding a uid/gid translation layer, so that the uid/gid for anything within an unprivileged container will be offset by a specified value, e.g., calls with user ID 1000 in a container are made to present to the host as user ID 1000000. This comes with its own host of issues which LXD tries to hide.

The short answer is that if security is any type of priority for the system in question and you want to run containerized processes, you should use an OS that implements container security directly in the kernel, like FreeBSD with jails or illumos with Zones, instead of depending on getting exactly the right configuration between all the moving pieces in the Linux container stack.

Re: Show HN: CLI tool for saving web pages as a single file

#175

Post by gambler »

derefr wrote: post #165
gambler wrote: post #150

Earlier quoted context omitted.

This is a good question. I think you can make it even better by generalizing the problem. How on earth do developers hope to advance general computing forward when simply running programs isn't a solved problem? Most software engineers I know don't run docker on their home PCs. What about people who aren't in IT? Does anyone here even care? The general attitude I see is "plebs don't need to run anything they can't ge…

But arbitrary programs are... arbitrary. Especially ones run by software engineers, and especially ones run by software engineers as part of a POSIX-alike “utility bag” ecosystem. Who’s to say that the user’s intent by running the program they just downloaded, isn’t to—say—overwrite a system folder? (Oh, wait, that’s exactly what Homebrew does, with the user’s full intent behind it!) There are tons of attempts to do…

>But what if you’re attempting to use those utilities against your real documents?

You copy or move documents inside the specific sandbox.

If you want a pipeline, you establish a chain of inbox/outbox folders.

Obviously, most of this should be done by the OS, not the user.

The workflow:

- You click "download" in your browser.

- When it's done and you click on your download, the OS asks how you want to open it. Instead of "execute" option you get "run in a sandbox" option.

- You type in the name of the sandbox, the app gets copies to /apps/sandboxName or something of that sort.

- The system automatically creates /apps/sandboxName/inbox and /apps/sandboxName/outbox.

- To process a file in some way, you drop it into inbox dir.

For command line, the only change would be switching from "executable pulls arbitrary files" to "I push specific file to the executable".

  zip -r squash.zip dir1
becomes

  | -r squash.zip | zip dir1 |
Start pipeline. Push squash.zip as an argument to zip, get the output. Zip would be the container name.

Re: Show HN: CLI tool for saving web pages as a single file

#176

Post by sah2ed »

donatzsky wrote: post #76
flatroze wrote: post #38

Earlier quoted context omitted.

That's it in the nutshell! It seems to work for basic pages quite well, I think that lazy load will work for most pages as long as the JavaScript is embedded (no -j flag provided) and the Internet connection is on. It saves what's there when the page is loaded, the rest is a gamble since every website implements infinite scroll differently. Authentication is another tricky part -- it's different for every browser. I…

For authentication, you could add an option for passing http headers, as well as accept Netscape-style cookie files. Whenever I want to download a video, using YouTube-dl, from a site that requires authentication, I first login using my browser and then exports the cookies using an extension.

May I ask what extension you use for cookie exporting?

Re: Show HN: CLI tool for saving web pages as a single file

#177

Post by sah2ed »

skinnymuch wrote: post #120
dredmorbius wrote: post #117

Earlier quoted context omitted.

FYI: "favorite" is one way of doing that through HN. Bookmarks, or downloads, externally.

Favorites is limited to a certain amount on HN before you start losing the oldest favorite.

Is this limitation documented somewhere?

After how many entries before the HN software started tripping on your favorites?

Re: Show HN: CLI tool for saving web pages as a single file

#178

Post by sansnomme »

mrieck wrote: post #157

If you only want a portion of a webpage I made a tool called SnipCSS for that: https://www.snipcss.com The desktop version saves an HTML file, stylesheet and images/fonts locally, and it only contains the HTML of the snippet with the CSS rules that apply to the DOM subtree of the element you select. I'm still working out bugs but it would be great if people try it out and let me know how it goes.

There have been quite a few extensions in this space:

https://stackoverflow.com/questions/10266334/add-on-to-copy-...

https://github.com/Dalimil/Web-Design-Pirate

Re: Show HN: CLI tool for saving web pages as a single file

#179

Post by sequoia »

FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

> Do you read all that code and evaluate the reputation of all dependencies?

Why of course. I do this for every piece of software on my computer, from the device drivers to the OS, I review every patch to firefox & chrome as well. /s

Running someone else's software inherently means extending them trust. This objection is especially confusing on a piece of software where you can actually inspect all the source if you like (unlike e.g. device drivers and OS code unless you run Gnu/Linux + all Free drivers, as few but RMS do).

Re: Show HN: CLI tool for saving web pages as a single file

#180

Post by mrieck »

sansnomme wrote: post #178
mrieck wrote: post #157

If you only want a portion of a webpage I made a tool called SnipCSS for that: https://www.snipcss.com The desktop version saves an HTML file, stylesheet and images/fonts locally, and it only contains the HTML of the snippet with the CSS rules that apply to the DOM subtree of the element you select. I'm still working out bugs but it would be great if people try it out and let me know how it goes.

There have been quite a few extensions in this space: https://stackoverflow.com/questions/10266334/add-on-to-copy-... https://github.com/Dalimil/Web-Design-Pirate

I tried SnappySnippet before when looking into the idea - it didn't work well for me and crashed often. I never saw DesignPirate, but just now I tried it and it didn't output any CSS. I'm not sure but it doesn't look like either of these use chrome.debugger API to call devtools api methods. (you get a warning in Chrome if you use that)

I'm hoping my tool will be better so it's good enough people would be willing to pay for it, but we'll just have to see.

Post reply on HN