Live data from Hacker News

ArchiveBox: Open-source self-hosted web archive

github.com

21–30 of 63 posts

Re: ArchiveBox: Open-source self-hosted web archive

#21
I actually talked to the author of ArchiveBox about 1-2 weeks ago. Nice guy and it's good that he's also friendly with the Internet Archive.

ArchiveBox uses WARC as it's backing store:

https://en.wikipedia.org/wiki/Web_ARChive

which is nice because it's standardized.

We were discussing integrating Polar web archives along with ArchiveBox and maybe having some sort of standard to automatically submit these WARCs to the Internet Archive as part of your normal browsing activity.

Polar has a similar web capture feature but it's not WARC

https://getpolarized.io/

(yet)...

WARC is probably the easiest standard for Polar to adopt. Right now we use HTML encoded in JSON objects.

When the user captures a web page we save all resources and store them in a PHZ file which you can keep as your own personal web archive.

What I'd like to eventually do is update our extension to auto-capture web pages so you could use Polar's cloud storage feature to basically store every page you've ever visited.

It really wouldn't be that much money per year. I did the math and it's about $50 per year to store your entire web history.

If I can get Polar over to WARC that would mean that tools like ArchiveBox and Polar could interop but we could do things like automatically send your documents you browse to the Internet Archive.

There's one huge problem though. What do we do about cookies and private data. I'm really not sure what to do there. It might be possible to strip this data for certain sites (news) without any risk of violating the users privacy.

Re: ArchiveBox: Open-source self-hosted web archive

#23
That's looks pretty nice! I have currently Wallabag running on a server (using it with a browser plugin to save pages) and it works pretty well. While Wallabag is strictly for websites as far as I understand, this seems to support more data (like bookmarks, history, etc.), which is great and I will certainly give it a try, luckily import from Wallabag is supported too.

Re: ArchiveBox: Open-source self-hosted web archive

#24

Great idea, but automated installer might not be ready for prime time. Automated install did not detect existing Python 3.6 on OSX, created quite the mess... Also, defaults to using google chrome vs. chromium when both are installed.

Add a github issue?

Re: ArchiveBox: Open-source self-hosted web archive

#26

Has anyone heard any news on Mozilla open sourcing Pocket? There are pieces that have been released but ultimately I would like to self host a Pocket archive similar to Wallabag

> Has anyone heard any news on Mozilla open sourcing Pocket?

All of Pocket? It's fairly straightforward to extract your Pocket data through their API (their export functionality leaves much to be desired unfortunately).

Re: ArchiveBox: Open-source self-hosted web archive

#28

I actually talked to the author of ArchiveBox about 1-2 weeks ago. Nice guy and it's good that he's also friendly with the Internet Archive. ArchiveBox uses WARC as it's backing store: https://en.wikipedia.org/wiki/Web_ARChive which is nice because it's standardized. We were discussing integrating Polar web archives along with ArchiveBox and maybe having some sort of standard to automatically submit these WARCs to th…

Why don't they store everything as plain documents in a zip file and keep metadata in a json? Seems more future proof/easier for users to manipulate than WARC

Re: ArchiveBox: Open-source self-hosted web archive

#29

I actually talked to the author of ArchiveBox about 1-2 weeks ago. Nice guy and it's good that he's also friendly with the Internet Archive. ArchiveBox uses WARC as it's backing store: https://en.wikipedia.org/wiki/Web_ARChive which is nice because it's standardized. We were discussing integrating Polar web archives along with ArchiveBox and maybe having some sort of standard to automatically submit these WARCs to th…

Why don't they store everything as plain documents in a zip file and keep metadata in a json? Seems more future proof/easier for users to manipulate than WARC

[deleted]

Re: ArchiveBox: Open-source self-hosted web archive

#30

I actually talked to the author of ArchiveBox about 1-2 weeks ago. Nice guy and it's good that he's also friendly with the Internet Archive. ArchiveBox uses WARC as it's backing store: https://en.wikipedia.org/wiki/Web_ARChive which is nice because it's standardized. We were discussing integrating Polar web archives along with ArchiveBox and maybe having some sort of standard to automatically submit these WARCs to th…

Why don't they store everything as plain documents in a zip file and keep metadata in a json? Seems more future proof/easier for users to manipulate than WARC

This is what Polar does but WARC is standardized... I think the argument against WARC though is that there isn't really much interchange. the point of standards is interchange IMO
Post reply on HN