Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

Post reply on HN
Print view View original

121–130 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#121

Post by jazzyjackson »

Apocryphon wrote: post #30
pzmarzly wrote: post #22

Earlier quoted context omitted.

Can you recommend some tools to manage mutable torrents? I.e. create them, edit them, download them and keep them downloaded up to date. BTW I recently tried using IPFS for a mutable public storage bucket and that didn't go well - downloads were very slow compared to torrents, and IPNS update propagation took ages. Perhaps torrents will do the job.

Man, looks like the first posts about IPFS cropped up on HN a decade ago. I remember seeing Neocities announcement of support for them. I wonder if that protocol has gotten anywhere since then.

There has been a large effort extended by Internet archive to adopt IPFS through their partnership with filecoin but IME the basic problems of the protocol remain - slow egress, slow discovery, someone still has to serve the file over a gateway to normie HTTP users...

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#122

Post by ikreymer »

TheTechRobo wrote: post #106
nikisweeting wrote: post #97

Earlier quoted context omitted.

Yes, this is true currently. If you need nice WARCs I recommend Browsertrix by our friends at Webrecorder instead. Its on my roadmap to improve this eventually, but currently I'm focused on saving raw files to a filesystem, because it's more accessible to most users, and easier to pipe into other tools. I encourage people to use ZFS to do deduping and compression at the filesystem layer.

Browsertrix (and Webrecorder tools in general) also violate the standard by modifying response data. It's supposed to be the raw bytes as they are sent over the network (minus TLS). The entire WARC ecosystem is kind of a mess.

This isn't really true, our tools do not just modify response data for no reason!

Our tools do the best that we can with an old format that is in use by many institutions. The WARC format does not account for H2/H3 data, which is used by most sites nowadays.

The goal of our (Webreocrder) tools is to preserve interactive web content with as much fidelity as possible and make them accessible/viewable in the browser. That means stripping TLS, H2/H3, sometimes forcing a certain video resolution, etc.. while preserving the authenticity and interactivity of the site. It can be a tricky balance.

If the goal is to preserve 'raw bytes sent over the network' you can use Wireshark / packet capture, but your archive won't necessarily be useful to a human.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#123

Post by ikreymer »

digitaldragon wrote: post #102
nikisweeting wrote: post #8

Earlier quoted context omitted.

I recommend Browsertrix for WARC creation, I think they are the best currently available for WARC/WACZ. ArchiveBox is also gearing up to support real cryptographic signing of archives using https://tlsnotary.org/ in an upcoming plugin. (in a way that actually solves the TLS non-repudation issue, which traditional "signing a WARC" does not, more info: https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd... )

Unfortunately, Browsertrix relies on the Chrome Devtools Protocol, which strips transfer encoding (and possibly transforms the data in other ways). This results in Browsertrix writing noncompliant WARC files, because the spec requires that the original transfer encoding be preserved.

Unfortunately, there is not much we can do about transfer-encoding, but the data is otherwise exactly as is returned from the browser. Browsertrix uses the browser to create web archives, so users get an accurate representation of what they see in their browser, which is generally what people want from archives.

We do the best we can with a limited standard that is difficult to modify. Archiving is always lossy, we try to reduce that as much as possible, but there are limits. People create web archives because they care about not losing their stuff online, not because they need an accurate record of transfer-encoding property in an HTTP connection. If storing the transfer-encoding is the most important thing, then yes, there are better tools for that.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#124

Post by manofmanysmiles »

jazzyjackson wrote: post #120
manofmanysmiles wrote: post #118

Earlier quoted context omitted.

Motivation: Have evidence that some content existed at a particular time. For example, let's say a major website publishes an article, and later they remove it, and there is no record of it ever existing. If I host an ArchiveBox, I can look at it and see "Oh here is that article. Looks line it was published after all." However, why should you believe me I didn't just make it up? If when I initially archived it, I com…

There's no reason to believe that the hashed and timestamped content was hosted at a particular domain, however (unless the content was signed by the author of course, then there's no Blockchain necessary). sure multiple peers could make some attestation that they saw it at that URL, but then you're back at square one of the reputation problem Internet archive as an institution with a reputation that holds up to a ju…

No, definitely not. I have no inherent reason to trust the people working at the Internet Archive over let's say close friend. For me trust is always a human to human concept, and no amount of tech or institutions will change that.

The more people I hear making a claim, the more I'm likely to deem the claim(s) as true. This is even true regarding the claims that cryptographic algorithms have the properties that make them useful in these contexts. I say this as someone who has even taken graduate level classes with Ron Rivest.

I'm not sure what will happen in a court. I imagine the more people that start making claims using cryptography as part of the supporting evidence, the more likely people will start to trust cryptography as a useful tool for resolving disputes about the veracity of claims.

So you would not get any value from multiple people making such claims?

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#125

Post by mikae1 »

nikisweeting wrote: post #72
mikae1 wrote: post #66

Earlier quoted context omitted.

Thanks for the box! Any examples of other possible really advanced features that might go for-pay? Is there any chance you will make current free features for-pay? That'd be rather off-putting for me as a home user.

No, everything currently free will stay free. The paid stuff currently is: - per-user permissions & groups - audit logging - auto CAPTCHA solving - burner credential management for FB/Insta/Twitter/etc. w/ auto phone based account verification ability - custom JS scripts for expanding comments, hiding pop ups, etc. - managed hosting + support Some of this stuff ^ is going to become free in upcoming releases, some wil…

Thanks for the clarification and thanks again for the great work!

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#126

Post by pabs3 »

404mm wrote: post #28

Somewhat similar topic, anyone has recommendations for a self-hosted internet website change monitoring system? I’ve been running Huginn for many years and it works well; however, I have a feeling the project is on its last leg. Also, it’s based on either text scraping (XPath/CSS/HTML and rss but it struggles with newer JS-based sites.

I recommend urlwatch, you run it from a terminal and send the output wherever you want, such as email via cron.

https://thp.io/2008/urlwatch/

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#127

Post by xiconfjs »

wongarsu wrote: post #16

Does this mean it's now possible to write plugins that dismiss cookie popups, solve captchas, scroll web pages etc.?

You mean ArchiveBox still doesn’t deal with cookie popups? If so, it’s quasi not useful for EU based web sites.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#128

Post by CorentinB »

ikreymer wrote: post #122
TheTechRobo wrote: post #106

Earlier quoted context omitted.

Browsertrix (and Webrecorder tools in general) also violate the standard by modifying response data. It's supposed to be the raw bytes as they are sent over the network (minus TLS). The entire WARC ecosystem is kind of a mess.

This isn't really true, our tools do not just modify response data for no reason! Our tools do the best that we can with an old format that is in use by many institutions. The WARC format does not account for H2/H3 data, which is used by most sites nowadays. The goal of our (Webreocrder) tools is to preserve interactive web content with as much fidelity as possible and make them accessible/viewable in the browser. Th…

He didn't say you modify the data for no reason, he said you violate the standard. Which is true. You could respect it, but you don't.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#129

Post by CorentinB »

ikreymer wrote: post #123
digitaldragon wrote: post #102

Earlier quoted context omitted.

Unfortunately, Browsertrix relies on the Chrome Devtools Protocol, which strips transfer encoding (and possibly transforms the data in other ways). This results in Browsertrix writing noncompliant WARC files, because the spec requires that the original transfer encoding be preserved.

Unfortunately, there is not much we can do about transfer-encoding, but the data is otherwise exactly as is returned from the browser. Browsertrix uses the browser to create web archives, so users get an accurate representation of what they see in their browser, which is generally what people want from archives. We do the best we can with a limited standard that is difficult to modify. Archiving is always lossy, we t…

You could use a proxy.

"Archiving is always lossy" No.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#130

Post by nikisweeting »

manofmanysmiles wrote: post #124
jazzyjackson wrote: post #120

Earlier quoted context omitted.

There's no reason to believe that the hashed and timestamped content was hosted at a particular domain, however (unless the content was signed by the author of course, then there's no Blockchain necessary). sure multiple peers could make some attestation that they saw it at that URL, but then you're back at square one of the reputation problem Internet archive as an institution with a reputation that holds up to a ju…

No, definitely not. I have no inherent reason to trust the people working at the Internet Archive over let's say close friend. For me trust is always a human to human concept, and no amount of tech or institutions will change that. The more people I hear making a claim, the more I'm likely to deem the claim(s) as true. This is even true regarding the claims that cryptographic algorithms have the properties that make…

I think the best solution is to have multiple people with reputation attest to the encrypted TLS content without being able to see the cleartext of it, that way they cant easily tamper with it.

See my comments on TLSNotary stuff below...

Post reply on HN