Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CLI tool for saving web pages as a single file

github.com

Post reply on HN
Print view View original

191–200 of 216 posts

Re: Show HN: CLI tool for saving web pages as a single file

#191

Post by lucideer »

baby wrote: post #155
lucideer wrote: post #122

Earlier quoted context omitted.

The capture includes JS, so this should work for most JS-dependent sites, with the exception of scripts loading other additional assets. Tbh, often those are superfluous, or egregious examples of bad web dev, so it seems a reasonable solution for most cases. SingleFile is a different approach, but it's a lot more involved/less convenient than a cli, and loading in something like WebDriver on the cli for this would be…

So any ajax and it won't work?

Any ajax won't work offline.

Ajax will* still work fine with an internet connection as long as those ajax endpoints don't require cookies and don't linkrot.

* not 100% sure how the tool handles relative URLs embedded in source : if it's not clever enough though, this is very fixable via PR (as in its not an architectural limitation)

Re: Show HN: CLI tool for saving web pages as a single file

#192

Post by baby »

lucideer wrote: post #191
baby wrote: post #155

Earlier quoted context omitted.

So any ajax and it won't work?

Any ajax won't work offline . Ajax will* still work fine with an internet connection as long as those ajax endpoints don't require cookies and don't linkrot. * not 100% sure how the tool handles relative URLs embedded in source : if it's not clever enough though, this is very fixable via PR (as in its not an architectural limitation)

What's the point of saving a webpage if it won't work offline?

Re: Show HN: CLI tool for saving web pages as a single file

#193

Post by flatroze »

mikaelmorvan wrote: post #94

The main problem with your code is that you only handle simple web1 site. What about javascript execution ? If you replay your capture, you have no idea of what you will see on general Web2 website. The only way I know to capture a web page properly is to "execute" it on a browser. Gildas, the guy behind SingleFile ( https://github.com/gildas-lormeau/SingleFile ) is well aware of that and his approach realy works eve…

Everything has its limits

Re: Show HN: CLI tool for saving web pages as a single file

#194

Post by MaxBarraclough »

roblabla wrote: post #148
masklinn wrote: post #106

Earlier quoted context omitted.

> I think that's an extremely uncharitable view on containers. It's an objective one. > There has been a massive amount of work put into securing containers for a variety of use cases using both layers available and by adding to the kernel. That doesn't change the fact that security was never the primary goal for containers, so secure containers were and are a bunch of tricks, kludges and prayers being built up in th…

Can you please quote how containers are not built with security in mind? What would even be the point of user namespacing, network namespaces, filesystem namespaces, etc... if not security?

Look at how the cloud providers offer support for containers.

Do they ever offer to run your container in the same VM as those of other customers?

They never do this. For secure isolation, they only trust VM isolation. It seems unlikely that this will change.

> What would even be the point of user namespacing, network namespaces, filesystem namespaces, etc... if not security?

They're for installation/configuration/administration. They allow you to run multiple applications on one Linux VM, and to configure them independently, almost as if you were running multiple VMs (with the advantage of lower overheads - only one instance of the kernel).

Kubernetes puts this to good use, letting you treat application deployments as commodities across your cluster.

Containers do not offer secure isolation. They are by nature much leakier than the isolation VMs can offer. The Docker folks still treat isolation-failures as bugs, of course. (Well, ignoring things like the way 'uptime' gives the uptime of the underlying machine, and not of your container.)

Re: Show HN: CLI tool for saving web pages as a single file

#195

Post by OJFord »

AnIdiotOnTheNet wrote: post #151
forgotmypw wrote: post #146

Earlier quoted context omitted.

Scripts I run have access to my development environment for a free software project I already keep in a public repo. What they don't have access to is my actual desktop, nor yesterday's snapshot of the VM desktop. It's obviously not as fast as running GNU on bare metal, but it's fast enough for text work.

> it's fast enough for text work. I guess this is where we set the bar in 2019.

Isn't the bar wherever you need it to be, on your own system, whatever the year?

I'm not GP commenter, but certainly what I do for a living is browse the the internet and edit text files.

Re: Show HN: CLI tool for saving web pages as a single file

#196

Post by MaxBarraclough »

yters wrote: post #162
roblabla wrote: post #148

Earlier quoted context omitted.

Can you please quote how containers are not built with security in mind? What would even be the point of user namespacing, network namespaces, filesystem namespaces, etc... if not security?

I recently took a course on how cgroups and namespaces work, and can be combined to create containers, and my impression is security is a huge kludge. For example, the capabilities are just a seeming random assortment of different permissions, with a big dumping ground in the admin capability. It's hard to see how such a system can be reliably secured. Plus, it's all open source with a couple core contributors. What'…

> my impression is security is a huge kludge

Docker itself could be called a huge kluge, at least compared to Solaris 'zones' and FreeBSD 'jails'.

They're similar to containers, but are supported directly by the kernel, whereas Docker has to pull together different kernel features to create its abstraction. [0]

> What's to stop some state agency inserting its code into the core? No way to review everything

1. This isn't a point about containers, it's a point about Free and Open Source software in general. Do you avoid all Open Source software when security matters? 2. I'm pretty sure the Linux kernel folks review everything, and I imagine the Docker folks do too 3. You're implicitly assuming that closed-source software is safe from government pressure. It is not.

[0] https://blog.jessfraz.com/post/containers-zones-jails-vms/

Re: Show HN: CLI tool for saving web pages as a single file

#197

Post by forgotmypw »

OJFord wrote: post #190
forgotmypw wrote: post #146

Earlier quoted context omitted.

Scripts I run have access to my development environment for a free software project I already keep in a public repo. What they don't have access to is my actual desktop, nor yesterday's snapshot of the VM desktop. It's obviously not as fast as running GNU on bare metal, but it's fast enough for text work.

Ah okay, that's why I thought to include 'at least for me' - because our 'all my development's were different! :) For me, the VM was my entire machine. (It wasn't meant to improve security, it was purely because I wanted Linux but couldn't have it on the host.)

That's a big part of the motivation for me.

I wanted GNU tools and environment, and I also wanted to not have to install Linux on the Apple hardware, because I don't have the patience for that.

Re: Show HN: CLI tool for saving web pages as a single file

#198

Post by forgotmypw »

AnIdiotOnTheNet wrote: post #151
forgotmypw wrote: post #146

Earlier quoted context omitted.

Scripts I run have access to my development environment for a free software project I already keep in a public repo. What they don't have access to is my actual desktop, nor yesterday's snapshot of the VM desktop. It's obviously not as fast as running GNU on bare metal, but it's fast enough for text work.

> it's fast enough for text work. I guess this is where we set the bar in 2019.

That's where my bar has been for 20+ years.

Re: Show HN: CLI tool for saving web pages as a single file

#199

Post by flatroze »

Avi-D-coder wrote: post #183
FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

Before I installed this tool, I checked if the author was a member of any well known organization and since they are not, I skimmed all the code for monolith and any of it's dependency that are not extremely popular, in this case mime-sniffer.

That dependency is now gone, thank you for the review

Re: Show HN: CLI tool for saving web pages as a single file

#200

Post by hollerith »

cookiecaper wrote: post #173
jolmg wrote: post #164

Earlier quoted context omitted.

> To me, this is kind of like saying you should just run stuff as root, because there might be a privelege escalation vulnerability which lets the code run as root anyway. But I never said it was preferable to run directly on the host. There are other choices. > My goal was to make things more secure, not completely secure. There is no such thing as completely secure. The argument against docker is more along the lin…

Linux relies on a concoction of properly-configured kernel subsystems to provide some level of isolation for containerized processes, and systems like LXD and Docker try to patch up the gaps. The cgroups interfaces don't offer much security stuff directly -- they're mainly about containing groups of process within certain resource consumption quotas, and afaik, don't really attempt to contemplate secure isolation dir…

I have no recent information, but about 10 years ago FreeBSD maintainers were telling people not to rely on jails for security.
Post reply on HN