Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox: Open-source self-hosted web archive

github.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: ArchiveBox: Open-source self-hosted web archive

#51

Post by mysterydip »

rakoo wrote: post #46
mysterydip wrote: post #37

Earlier quoted context omitted.

One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.

Very true, but I'd say SQlite is the one application dependency that could have a pass: - they promise support until at least 2050 ( https://sqlite.org/lts.html ) - if a promise isn't enough, SQlite is to be supported for the entire lifetime of the Airbus A350 airframe ( https://mobile.twitter.com/copiousfreetime/status/6758345433... ). I would assume Airbus will be paying for it. A real problem with application depe…

Also excellent points I waan't aware of, thanks!

Re: ArchiveBox: Open-source self-hosted web archive

#52

Post by nikisweeting »

joustfawrthis wrote: post #20

Great idea, but automated installer might not be ready for prime time. Automated install did not detect existing Python 3.6 on OSX, created quite the mess... Also, defaults to using google chrome vs. chromium when both are installed.

Yup, I totally agree (I'm the creator @pirate on Github).

I personally don't like automated setup scripts, which is one of the reasons I spent a bunch of time on our Manual Install docs: https://github.com/pirate/ArchiveBox/wiki/Install

Pip, apt, & homebrew packages are the install methods we're moving towards currently. I just caved to user demand in the short-term and added a helper script about a year ago as a crutch until we finished releasing the real packages.

Stay tuned for updates on that progress here: https://github.com/pirate/ArchiveBox/issues/120

Re: ArchiveBox: Open-source self-hosted web archive

#53

Post by nikisweeting »

burtonator wrote: post #21

I actually talked to the author of ArchiveBox about 1-2 weeks ago. Nice guy and it's good that he's also friendly with the Internet Archive. ArchiveBox uses WARC as it's backing store: https://en.wikipedia.org/wiki/Web_ARChive which is nice because it's standardized. We were discussing integrating Polar web archives along with ArchiveBox and maybe having some sort of standard to automatically submit these WARCs to th…

Hi Kevin!

Thanks for chiming in here! I only just saw ArchiveBox hit HN while I was away for the weekend. Just responding to everything now...

(@everyone in this thread, go check out Polar, it's awesome!)

Re: ArchiveBox: Open-source self-hosted web archive

#54

Post by nikisweeting »

walterbell wrote: post #3

If this were running on a self-hosted server, could it be invoked from an iOS shortcut on phone/tablet?

Yup, I actually do this right now using Shortcuts.app on my phone to SSH into my server and pipe my latest bookmarks from Pocket into ArchiveBox!

It's a nice way to update my archive with one tap from my phone.

Re: ArchiveBox: Open-source self-hosted web archive

#56

Post by nikisweeting »

ausjke wrote: post #27

what about SPA sites that javascript renders the html, if this can archive SPA site that will be great.

The short answer is that it can (usually) archive SPAs sucessfully. The long answer is that the wget archive method will download the JS, and in theory it should execute in the archived version just like it does on the normal site, but in practice it doesn't always work 100%. Luckily that's why ArchiveBox also saves a Chrome-rendered version of the page as a PDF, Screenshot, and DOM dump, so you should be able to archive most javascript-heavy content without too many problems.

Re: ArchiveBox: Open-source self-hosted web archive

#57

Post by nikisweeting »

thrilleratplay wrote: post #25

Has anyone heard any news on Mozilla open sourcing Pocket? There are pieces that have been released but ultimately I would like to self host a Pocket archive similar to Wallabag

I love Mozilla and I'd support them open sourcing it. But I think they probably don't have much motivation to do so, as there are plenty of competing archiving tools already in the public domain, and their value-add as a paid archiving service would be reduced if anyone could run a white-label version of Pocket with one click.

Re: ArchiveBox: Open-source self-hosted web archive

#58

Post by walterbell »

nikisweeting wrote: post #54
walterbell wrote: post #3

If this were running on a self-hosted server, could it be invoked from an iOS shortcut on phone/tablet?

Yup, I actually do this right now using Shortcuts.app on my phone to SSH into my server and pipe my latest bookmarks from Pocket into ArchiveBox! It's a nice way to update my archive with one tap from my phone.

Could the intermediate step via Pocket be avoided, e.g. could the URL be passed directly from iOS to ArchiveBox?

Re: ArchiveBox: Open-source self-hosted web archive

#59

Post by nikisweeting »

toomuchtodo wrote: post #42
progval wrote: post #34

Earlier quoted context omitted.

Depends on what you want to use your archive for. If it's just for opening it locally in a web browser, mirroring the content is enough.

Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.

We intend to store both the WARC of the headless broswer render and the wget clone. (I'm the creator @pirate on Github). The idea is to archive anything in many redundant formats (HTML clone, WARC, PDF, etc) and for people to run this on a deduplicated filesystem like ZFS/BTRFS or disable specific methods if they care about saving space.

You can track the progress on implementing WARC saving for all archived resources here: https://github.com/pirate/ArchiveBox/issues/130

Re: ArchiveBox: Open-source self-hosted web archive

#60

Post by nikisweeting »

walterbell wrote: post #58
nikisweeting wrote: post #54

Earlier quoted context omitted.

Yup, I actually do this right now using Shortcuts.app on my phone to SSH into my server and pipe my latest bookmarks from Pocket into ArchiveBox! It's a nice way to update my archive with one tap from my phone.

Could the intermediate step via Pocket be avoided, e.g. could the URL be passed directly from iOS to ArchiveBox?

Yes, easily. Just append the URL to a file with SSH and pipe the file into ArchiveBox, or pipe the url directly into ArchiveBox via stdin.
Post reply on HN