Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

Post reply on HN
Print view View original

101–110 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#101

Post by A4ET8a8uTh0 »

nikisweeting wrote: post #100
A4ET8a8uTh0 wrote: post #88

Those additions are welcome, but if I could request one -- I and one that it is very consistently requested -- feature: - backing up an entire page Yes, it is hard. Yes, for non-pure html pages is extra kind of painful, but that would honestly making archivebox go from nice to have to.. yes, I have an actual archive I can use when stuff goes down.

Do you mean backing up an entire domain? Like example.com/* If so that's starting to roll out in v0.8.5rc50, check out the archivebox/crawls/ folder. If you mean archiving a single page more thoroughly, what do you find is missing in Archivebox? Are you able to get singlefile/chrome/wget html when archiving?

Edit: The first option. ( previous stuff removed )

Lemme check my current version ( edit: 0.7.2 -- ty, I will update and test soon :D)

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#102

Post by digitaldragon »

nikisweeting wrote: post #8
toomuchtodo wrote: post #5

https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to ena…

I recommend Browsertrix for WARC creation, I think they are the best currently available for WARC/WACZ. ArchiveBox is also gearing up to support real cryptographic signing of archives using https://tlsnotary.org/ in an upcoming plugin. (in a way that actually solves the TLS non-repudation issue, which traditional "signing a WARC" does not, more info: https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd... )

Unfortunately, Browsertrix relies on the Chrome Devtools Protocol, which strips transfer encoding (and possibly transforms the data in other ways). This results in Browsertrix writing noncompliant WARC files, because the spec requires that the original transfer encoding be preserved.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#103

Post by nikisweeting »

bigiain wrote: post #90
nikisweeting wrote: post #17

Earlier quoted context omitted.

> The default settings should be "safe" for the default user, I 100% agree, but because private archiving is doable but NOT 100% safe yet I cant make that mode the default. The difficult reality currently is that archiving anything non-public is not simple to make safe. Every capture will contain reflected session cookies, usernames, and PII, and other sensitive content. People don't understand that this means if the…

That's a really good response, thanks. I've been very impressed by all of your responses in here, but that one in particular shows empathy, compassion, and a deep deep subject matter expertise.

Thank you. And thank you for taking the time to read all of it, there's a lot of great questions being asked.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#104

Post by nikisweeting »

A4ET8a8uTh0 wrote: post #101
nikisweeting wrote: post #100

Earlier quoted context omitted.

Do you mean backing up an entire domain? Like example.com/* If so that's starting to roll out in v0.8.5rc50, check out the archivebox/crawls/ folder. If you mean archiving a single page more thoroughly, what do you find is missing in Archivebox? Are you able to get singlefile/chrome/wget html when archiving?

Edit: The first option. ( previous stuff removed ) Lemme check my current version ( edit: 0.7.2 -- ty, I will update and test soon :D)

Ah ok. One caveat: it's only available via the 'archivebox shell' / Python API currently, the CLI & web UIs for full depth crawling will come later.

You can play around with the models and tasks, but I would wait a few weeks for it to stabilize and check again, it's still under heavy active development

Check archivebox/archivebox:dev periodically

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#106

Post by TheTechRobo »

nikisweeting wrote: post #97
pabs3 wrote: post #95

Unfortunately ArchiveBox uses wget, so it produces non-standard WARC files. Sadly there are lots of things like this in the WARC ecosystem. https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem

Yes, this is true currently. If you need nice WARCs I recommend Browsertrix by our friends at Webrecorder instead. Its on my roadmap to improve this eventually, but currently I'm focused on saving raw files to a filesystem, because it's more accessible to most users, and easier to pipe into other tools. I encourage people to use ZFS to do deduping and compression at the filesystem layer.

Browsertrix (and Webrecorder tools in general) also violate the standard by modifying response data. It's supposed to be the raw bytes as they are sent over the network (minus TLS).

The entire WARC ecosystem is kind of a mess.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#107

Post by A4ET8a8uTh0 »

nikisweeting wrote: post #104
A4ET8a8uTh0 wrote: post #101

Earlier quoted context omitted.

Edit: The first option. ( previous stuff removed ) Lemme check my current version ( edit: 0.7.2 -- ty, I will update and test soon :D)

Ah ok. One caveat: it's only available via the 'archivebox shell' / Python API currently, the CLI & web UIs for full depth crawling will come later. You can play around with the models and tasks, but I would wait a few weeks for it to stabilize and check again, it's still under heavy active development Check archivebox/archivebox:dev periodically

No worries. I can do that.

You guys probably hear it all the time, but you are doing lords work. If I thought I could be of use in that project, I would be trying to contribute myself ( in fact, let me see if there a way I can participate in a useful manner ).

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#109

Post by shiroiushi »

bityard wrote: post #92

So, after reading through the comments and website, I just realized I used ArchiveBox a month or two ago for a very specific purpose. You see, I inherited a boat. This boat belonged to my father. He was not materialistic but he took very good care of the things he cared about, and he cared about this boat. It's an old 18' aluminum fishing/cruising boat built in the early 1960's. It's not particularly valuable as a co…

>because 10 or 20 years ago, there were quite a few active web forums containing informational/tutorial threads from the proud owners of these old boats. ... But the forums are gone, so a large chunk of knowledge on these boats is too, probably forever.

These days, that kind of info would be locked up in a closed Discord chat somewhere, so you can forget about people 20 years from now ever seeing it.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#110

Post by petertodd »

nikisweeting wrote: post #73
jasonfarnon wrote: post #69

Earlier quoted context omitted.

I always wonder about this when someone gets in hot water based on something on the wayback machine and the person says the archive was tampered with. Can you elaborate on "prove that the archive was created in the past, prior to there being a reason to tamper it"? What exactly does opentimestamps certify?

OpenTimestamps alone can not currently prove anything because TLS session keys are symmetric. The client can forge anything and attest to it falsely. Unless you 100% trust the archiver (in which case you can trust their timestamps), you need TLSNotary or another reputable third party in the loop as a bare minimum. But more critically: currently the legal standard for evidence is... screenshots. We have a lot of educa…

> OpenTimestamps alone can not currently prove anything because TLS session keys are symmetric.

Timestamps can prove that the data existed prior to there being a known reason to modify it. While that's not as good as direct signing, that's often still enough to be very useful. The statement that OTS "can not currently prove anything" is incorrect.

A really good example of this is the Hunter Biden email verification. I used OpenTimestamps to prove that the DKIM key that signed the email was in fact used by Google at the time, by providing a Google-signed email that had been timestamped years ago: https://github.com/robertdavidgraham/hunter-dkim/tree/main/o...

That's convincing evidence, because it's highly implausible that I would have been working to fake Hunter's emails years before they even came up as an election issue.

Post reply on HN