Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

161–166 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#161

https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to ena…

I recommend Browsertrix for WARC creation, I think they are the best currently available for WARC/WACZ. ArchiveBox is also gearing up to support real cryptographic signing of archives using https://tlsnotary.org/ in an upcoming plugin. (in a way that actually solves the TLS non-repudation issue, which traditional "signing a WARC" does not, more info: https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd... )

>ArchiveBox is also gearing up to support real cryptographic signing of archives

That's a really interesting point. The gut reaction is "why are we wasting time on adding a nice-to-have such as a very fancy cousin of the MD5 checksum when the real meat of the time & effort is maximizing data download and scale"

But then, then go read the book 1984 and it may become clear the importance of ensuring the data is unchanged down the road.

But if this is a hedge against hypothetical future 1984 world, one would have to ask - what if the only file available has the wrong md5sum? Because then most people would say, "welp, something is better than nothing" and that's it. Perhaps something that might provide additional information about what/how/where something was changed in more detail.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#162
post #143

Earlier quoted context omitted.

Yep. That's what I've built for myself, I just can't really get at the data inside ArchiveBox until I upgrade.

How did you build it? I can imagine an architecture where I throw everything into ArchiveBox, then run VectorDB as a plugin with Gradio or some such as the client. https://vectordb.com/

You're overcomplicating things. You don't need a vector database, FTS works just as well for non-homogenous content.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#163

As someone who was archiving a doomed website earlier today using wget, I was reminded that really need to get ArchiveBox working... I used to rely on my Pinboard subscription, but apparently archive exports haven't worked for years, so those days are over.

FYI I am able to export (as JSON/HTML/XML) my Pinboard bookmarks.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#164
post #147

Earlier quoted context omitted.

It does, you just have to set up a chrome profile that has an extension to hide cookie popups, or use a profile where you've already accepted/closed them and have a session. You can archive with any chrome profile with arbitrary extensions enabled, so you can use uBlock, I still Don't care about cookies, Ghostery, etc.

How do you set this up? I found this relevant issue[1] but it doesn't explain how to get it working. [1] https://github.com/ArchiveBox/ArchiveBox/issues/211

https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Insta...

https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration/...

https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv...

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#165

@nikisweeting Is abx-dl already available or is it coming? I took a quick dive and didn't see a repo under the org. I'm happy to help package this up once it is available.

Not currently available, it should be out soon after v0.9 is released. Currently `mkdir tmp_data && cd tmp_data; archivebox install; archivebox add ...` is effectively equivalent to what `abx-dl` will do.

Check here for progress: https://github.com/ArchiveBox/abx-dl

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#166

Earlier quoted context omitted.

I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution. https://addons.mozilla.org/en-US/firefox/addon/single-file/

Singlefile is excellent, Gildas is a great developer. ArchiveBox has had singlefile as one of its extractors built in for years :)

I just installed the ArchiveBox add-on to Firefox, but I'm not sure how to find the page that I just archived. Do people normally run their own server to collect the pages?
Post reply on HN