Show HN: CLI tool for saving web pages as a single file • Hacker News Board

Re: Show HN: CLI tool for saving web pages as a single file

#131

Post by tinsx » Fri, Aug 23, 2019, 1:02 PM UTC

mikaelmorvan wrote: post #128

lucideer wrote: post #122

Earlier quoted context omitted.

The capture includes JS, so this should work for most JS-dependent sites, with the exception of scripts loading other additional assets. Tbh, often those are superfluous, or egregious examples of bad web dev, so it seems a reasonable solution for most cases. SingleFile is a different approach, but it's a lot more involved/less convenient than a cli, and loading in something like WebDriver on the cli for this would be…

superfluous, or egregious examples of bad web dev?? Do you know what Web 2.0 is? Do you know what are React, Angular, and the other JS Frameworks? When you create a modern webapp, a lot of data are retrieved from servers as Json and formated in the browser in Javascript. Even sometimes Css is generated on browser-side. Even more, on webapp where user login is taken into account, the display is modified accordingly. T…

I think that's exactly what that person means by superfluous and egregious examples of bad web development; SPAs, javascript frameworks of that nature. :p

Re: Show HN: CLI tool for saving web pages as a single file

#132

Post by chiefalchemist » Fri, Aug 23, 2019, 1:05 PM UTC

h1d wrote: post #123

mikaelmorvan wrote: post #94

The main problem with your code is that you only handle simple web1 site. What about javascript execution ? If you replay your capture, you have no idea of what you will see on general Web2 website. The only way I know to capture a web page properly is to "execute" it on a browser. Gildas, the guy behind SingleFile ( https://github.com/gildas-lormeau/SingleFile ) is well aware of that and his approach realy works eve…

Is there a CLI version of it?

Does it also capture tracking crap? Wouldn't it be great to grab a URL via CL and read it local without anyone knowing?

If it does capture the tracking shite, would disconnecting from the internet be a good enough blocker?

Re: Show HN: CLI tool for saving web pages as a single file

#133

Post by mikaelmorvan » Fri, Aug 23, 2019, 1:07 PM UTC

tinsx wrote: post #131

mikaelmorvan wrote: post #128

Earlier quoted context omitted.

superfluous, or egregious examples of bad web dev?? Do you know what Web 2.0 is? Do you know what are React, Angular, and the other JS Frameworks? When you create a modern webapp, a lot of data are retrieved from servers as Json and formated in the browser in Javascript. Even sometimes Css is generated on browser-side. Even more, on webapp where user login is taken into account, the display is modified accordingly. T…

I think that's exactly what that person means by superfluous and egregious examples of bad web development; SPAs, javascript frameworks of that nature. :p

Yes, the debate between building a SPA with rich features or Web old pages with good SEO is eternal :) We see more and more an hybrid approach that can be called web 1.5 :)

Re: Show HN: CLI tool for saving web pages as a single file

#134

Post by mikekchar » Fri, Aug 23, 2019, 1:10 PM UTC

flatroze wrote: post #90

mikekchar wrote: post #58

With respect to the Unlicense, does anybody have any knowledge about how good it is in countries which don't allow you to intentionally pass things into the public domain (most countries that aren't the US)? How does it compare to CC0 in that respect?

Which license would you recommend to release this software under to reach the broad adoption yet permissive terms, if not the Unlicense?

I honestly don't know. That's why the question :-) Is CC0 good for software? It seems to be a bit more complete from a non-US view point, but I don't know if there are lurking situations. Possibly MIT is better -- it's pretty darn permissive. I'm really just soliciting opinions.

Re: Show HN: CLI tool for saving web pages as a single file

#135

Post by gildas » Fri, Aug 23, 2019, 1:12 PM UTC

chiefalchemist wrote: post #132

h1d wrote: post #123

Earlier quoted context omitted.

Is there a CLI version of it?

Does it also capture tracking crap? Wouldn't it be great to grab a URL via CL and read it local without anyone knowing? If it does capture the tracking shite, would disconnecting from the internet be a good enough blocker?

By default, it will capture everything that would be interpreted/displayed. However, you can also run it with Selenium (and Firefox) instead of Puppeteer, and install your favorite extensions to block all the unwanted resources (there's a switch for that).

Re: Show HN: CLI tool for saving web pages as a single file

#136

Post by wolco » Fri, Aug 23, 2019, 1:18 PM UTC

mikaelmorvan wrote: post #128

lucideer wrote: post #122

Earlier quoted context omitted.

The capture includes JS, so this should work for most JS-dependent sites, with the exception of scripts loading other additional assets. Tbh, often those are superfluous, or egregious examples of bad web dev, so it seems a reasonable solution for most cases. SingleFile is a different approach, but it's a lot more involved/less convenient than a cli, and loading in something like WebDriver on the cli for this would be…

superfluous, or egregious examples of bad web dev?? Do you know what Web 2.0 is? Do you know what are React, Angular, and the other JS Frameworks? When you create a modern webapp, a lot of data are retrieved from servers as Json and formated in the browser in Javascript. Even sometimes Css is generated on browser-side. Even more, on webapp where user login is taken into account, the display is modified accordingly. T…

Sometimes we spent too much time in our own ecosystem. In 2019 most sites globally still use php and jquery.

Web 2.0 refers to the use of ajax. This refers to the early 2000s sajax, jquery..

If you want to separate angular, react, vue maybe it's web 3.0.. but wasn't web 3.0 referred to as mobile?

Re: Show HN: CLI tool for saving web pages as a single file

#137

Post by wolco » Fri, Aug 23, 2019, 1:21 PM UTC

mike-cardwell wrote: post #103

FreeHugs wrote: post #61

One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…

These are the install instructions the docs say to use: $ git clone https://github.com/Y2Z/monolith.git $ cd monolith $ cargo install These are the ones I used: $ git clone https://github.com/Y2Z/monolith.git $ cd monolith $ sudo docker run --rm -w "$(pwd)" -v "$(pwd):$(pwd)" -u "$(id -u):$(id -g)" rust cargo install That isolated the build process. Similar method to isolate the execution of the built project: $ cd t…

For someone having trouble with npm and windows 10, the suggestion couldn't come at a better time.

Re: Show HN: CLI tool for saving web pages as a single file

#138

Post by lucideer » Fri, Aug 23, 2019, 1:21 PM UTC

mikaelmorvan wrote: post #128

lucideer wrote: post #122

Earlier quoted context omitted.

The capture includes JS, so this should work for most JS-dependent sites, with the exception of scripts loading other additional assets. Tbh, often those are superfluous, or egregious examples of bad web dev, so it seems a reasonable solution for most cases. SingleFile is a different approach, but it's a lot more involved/less convenient than a cli, and loading in something like WebDriver on the cli for this would be…

superfluous, or egregious examples of bad web dev?? Do you know what Web 2.0 is? Do you know what are React, Angular, and the other JS Frameworks? When you create a modern webapp, a lot of data are retrieved from servers as Json and formated in the browser in Javascript. Even sometimes Css is generated on browser-side. Even more, on webapp where user login is taken into account, the display is modified accordingly. T…

With the exception of actual XHR requests (which should ideally be for dynamic resources, and as such somewhat outside the remit of saving a webpage), I was referring specifically to JS loading JS, etc. solutions. React, Angular do not recommend/advise you to do this. This isn't a requirement in Web 2.0 or Web 5.0 or anything else.

In terms of React at least, fetch requests are not a part of the framework in any way and any present would typically be done in custom code in lifecycle methods. Even Redux, is—by default—client-side only. Stores are in-memory, actions populating them would make fetch requests with React/Redux-independent logic.

Other JS frameworks are, typically, the same. And all of that is just considering dynamic XHR. Loading scripts is much less typical, and never required. The most common application of this I've seen is the GA snippet, which mainly does it to ensure the load is async without relying on developer implementation: it's 100% unnecessary to do it this way.

So yes, unless you're distributing a tracking snippet that you expect non-devs to be blindly pasting into their wordpress panels and still have it work efficiently, generally speaking use of this method is never necessary, and commonly a red flag for poor architecture.

Re: Show HN: CLI tool for saving web pages as a single file

#139

Post by UI_at_80x24 » Fri, Aug 23, 2019, 1:23 PM UTC

chiefalchemist wrote: post #130

mikaelmorvan wrote: post #94

The main problem with your code is that you only handle simple web1 site. What about javascript execution ? If you replay your capture, you have no idea of what you will see on general Web2 website. The only way I know to capture a web page properly is to "execute" it on a browser. Gildas, the guy behind SingleFile ( https://github.com/gildas-lormeau/SingleFile ) is well aware of that and his approach realy works eve…

I hear ya but another way to look at this is... The main problem with too many websites is they've become too much about technology and have left visitors, as well as the spirit and intent of the internet behind.

It stopped being about information and started being entertainment.

I agree with you. BadSite: "we want you to experience this.."

GoodSite: "we want you to learn this..."

Re: Show HN: CLI tool for saving web pages as a single file

#140

Post by mxuribe » Fri, Aug 23, 2019, 1:39 PM UTC

fit2rule wrote: post #86

I've been printing to PDF for decades now, and nothing comes close to the ease of use and versatility of 2 decades worth of interesting web pages .. I have pretty much every interesting article, including many from HN, from decades of this habit. Need to find all articles relating to 'widget'? $ ls -l ~/PDFArchive/ | grep -i widget This has proven so valuable, time and again .. there is a great joy in not having to m…

Its funny, on a rare occasion, I too have saved some content as a PDF - more so for archiving rather than for offline viewing...But i guess i never thought to scale it for all/most of my bookmarks. It seems so obvious now after reading your comment. However, my experience with PDFs has been negative. From filesize to slow booting of myriad pdf viewers, etc., it just seems like viewing stuff in native html, text is better - at least for what I've experienced. Further, my preferred browser - firefox - leaves much to be desired in this arena of generating proper PDFs, and i end up switching to chrome (bleh!) just to "PDF something" that i saw/read online. Again, this function in firefox is not something that i use as often, hence why i stick with FF, and not gone back to chrome. However, going back to your approach....I wonder if i can use a tool - either like this monolith or singlefile, or even pupeteer, etc. - to snapshot web content, but save it into html instead of pdf. I would guess html content is still grep-able (as you noted for your PDF local searches). Hmmm...a local cache of my own offline bookmarks...Hmmm, interesting. Thank you for this inspiration!!