Note that SingleFile can easily run on command line too, cf. https://github.com/gildas-lormeau/SingleFile/tree/master/cli.
Show HN: CLI tool for saving web pages as a single file
91–100 of 216 posts
Re: Show HN: CLI tool for saving web pages as a single file
#92Earlier quoted context omitted.
I just clicked on the OP user name first which has the caption Bad boy . Then I read your comment. Now I worry more about the security aspect than before.
Its funny how we sometimes think about risk; if the caption was 'good egg' would you be more trusting?
I’d have a different initial response. Somehow it triggered my brain to think more about security and be on alert. I’m any case this faded away quickly. The mind is beautiful :)
Re: Show HN: CLI tool for saving web pages as a single file
#93I really like this concept, and I've been using an npm package called inliner which does this too: https://www.npmjs.com/package/inliner I'm glad there's more people taking a look at the use case, and I'd be interested to see a list of similar solutions. If you combine this with Chrome's headless mode, you can prerender many pages that use JavaScript to perform the initial render, and then once you're done send it to…
You want this: https://github.com/gildas-lormeau/SingleFile/tree/master/cli.
Re: Show HN: CLI tool for saving web pages as a single file
#94The main problem with your code is that you only handle simple web1 site.
What about javascript execution ? If you replay your capture, you have no idea of what you will see on general Web2 website.
The only way I know to capture a web page properly is to "execute" it on a browser.
Gildas, the guy behind SingleFile (https://github.com/gildas-lormeau/SingleFile) is well aware of that and his approach realy works everytime.
Try on a Facebook post, a Tweet, ... It just works.
Re: Show HN: CLI tool for saving web pages as a single file
#95This is awesome. One question though: how does it handle the same resource (e.g. image) appearing multiple times? Does it store multiple copies, potentially blowing up the file size? If not, how does it link to them in a single HTML file? If or if so, is there any way to get around it without using MHTML (or have you considered using MHTML in that case)? Also, side-question about Rust: how do I get rid of absolute fi…
SingleFile which can run on CLI too handles this by using CSS variables.
Re: Show HN: CLI tool for saving web pages as a single file
#96Earlier quoted context omitted.
Thank you! I'll add it as an issue, since it could definitely be useful for "archiving" certain resources more than 1 level deep. Do you remember the name of that tool by any chance?
I wish I could remember the exact tool, this was over a decade ago. If you just do a quick internet search for such a tool you'll likely find whatever I used, it certainly wasn't anything sophisticated. It was a Windows GUI tool designed specifically for the task. Something makes me think that 'GetRight' tool might have been able to do the same thing, but I can't seem to see the feature on their website.
yes, "teleport pro" for win98, you could scrape a site and duplicate locally or scrape only for specific file type or size, had recursive link follow depth option and created several threads for the requests(sniff)
Re: Show HN: CLI tool for saving web pages as a single file
#97Earlier quoted context omitted.
Its funny how we sometimes think about risk; if the caption was 'good egg' would you be more trusting?
I’d have a different initial response. Somehow it triggered my brain to think more about security and be on alert. I’m any case this faded away quickly. The mind is beautiful :)
There is nothing either girl or boy, but thinking makes it so.
Re: Show HN: CLI tool for saving web pages as a single file
#98One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…
I often run new things in a vm with snapshots. When I don’t it’s as a limited user.
Re: Show HN: CLI tool for saving web pages as a single file
#99MHTML is pretty good for this already btw (not to take away from this neat project though :)). Similarly stores assets as base64'd data URIs and saves it as a single file. Can be enabled in Blink-based browsers using a settings flag and previously in Firefox using addons (also in the past natively in Opera and IE).
One issue with MHTML is that it does not seem to be currently supported by iframes. The use case I was working on was comparing search results from Google and DuckDuckGo by simply scraping and downloading to later embed. For that, I used a cli tool from an open source library [1]. MHTML seems like a nice format but I'm not sure if there's a library to convert them into stand-alone HTML files. [1] https://github.com/g…
> One issue with MHTML is that it does not seem to be currently supported by iframes.
There no needs to insert MHTML page into IFrame!
If you need insert MHTML content into IFrame, just convert it to HTML+JS firstly.
Re: Show HN: CLI tool for saving web pages as a single file
#100One thing I always wonder when I see native software posted here: How do you guys handle the security aspect of executing stuff like this on your machines? Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies? Do you execute it in a sandboxed environment? Do you just hope for the…
> Skimming the repo it has about a thousand lines of code and a bunch of dependencies with hundreds of sub-dependencies. Do you read all that code and evaluate the reputation of all dependencies?
I usually do a quick check to see if there's any red flag. Like URLs or Base64 blobs. I also try to stay away from programs written in languages whose environment I don't know so I can check if any dependency stands out wrt what the program claims to do.
> Do you execute it in a sandboxed environment?
Either I run it on an untrusted server, or on my laptop as an unpriviliged user (with no access to X/wayland).