Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

Post reply on HN
Print view View original

131–140 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#131

Post by nikisweeting »

CorentinB wrote: post #129
ikreymer wrote: post #123

Earlier quoted context omitted.

Unfortunately, there is not much we can do about transfer-encoding, but the data is otherwise exactly as is returned from the browser. Browsertrix uses the browser to create web archives, so users get an accurate representation of what they see in their browser, which is generally what people want from archives. We do the best we can with a limited standard that is difficult to modify. Archiving is always lossy, we t…

You could use a proxy. "Archiving is always lossy" No.

You're talking to the guy who built the best proxy recorder in the archiving industry ;) ikreymer created https://pywb.readthedocs.io/en/latest/

I think he has more context than any of us on the limits of proxy archiving vs browser based archiving.

But also if you really need perfect packet-level replication, just wireshark it as he said. Why bother with WARCs at all?

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#132

Post by nikisweeting »

CorentinB wrote: post #128
ikreymer wrote: post #122

Earlier quoted context omitted.

This isn't really true, our tools do not just modify response data for no reason! Our tools do the best that we can with an old format that is in use by many institutions. The WARC format does not account for H2/H3 data, which is used by most sites nowadays. The goal of our (Webreocrder) tools is to preserve interactive web content with as much fidelity as possible and make them accessible/viewable in the browser. Th…

He didn't say you modify the data for no reason, he said you violate the standard. Which is true. You could respect it, but you don't.

imo the Webrecorder stuff is truly state of the art, if they're pushing the limits of WARC standards it's for good reason, and I trust their judgement. They pioneered the newer WACZ standard and are really pushing the whole field forward.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#133

Post by nikisweeting »

petertodd wrote: post #111
nikisweeting wrote: post #64

Earlier quoted context omitted.

We're going to add TLSNotary support for real cryptographic signing, see my comments below :) Timestamping is also on my roadmap, definitely as a plugin (and likely paid) as it's more corporate users that really need it. We need to keep some of the really advanced attestation features paid to be able to support the rest of the business.

> We're going to add TLSNotary support for real cryptographic signing, see my comments below :) Last I checked TLSNotary requires a trusted third party. I would strongly suggest timestamping TLSNotary evidence, to be able to prove that evidence was created prior to any of these trusted third parties being compromised.

Of course, TLSNotary stuff would necessarily come with a whole ecosystem, including some sort of transparency log like certificate transparency logs, DNS record keeping, timestamping, etc.

But we'll start with the basics and work our way up to completeness.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#134

Post by nikisweeting »

petertodd wrote: post #110
nikisweeting wrote: post #73

Earlier quoted context omitted.

OpenTimestamps alone can not currently prove anything because TLS session keys are symmetric. The client can forge anything and attest to it falsely. Unless you 100% trust the archiver (in which case you can trust their timestamps), you need TLSNotary or another reputable third party in the loop as a bare minimum. But more critically: currently the legal standard for evidence is... screenshots. We have a lot of educa…

> OpenTimestamps alone can not currently prove anything because TLS session keys are symmetric. Timestamps can prove that the data existed prior to there being a known reason to modify it. While that's not as good as direct signing, that's often still enough to be very useful. The statement that OTS "can not currently prove anything" is incorrect. A really good example of this is the Hunter Biden email verification.…

Ok, fair point, they prove that content existed at some point in time, which is useful sometimes. But I don't want people to over-rely on that as "good enough", we can do much better, it's too low a bar for a whole ecosystem of archiving to rely on when we now have a viable solution to fix it (TLSNotary or others).

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#135

Post by nikisweeting »

newman314 wrote: post #105

@nikisweeting Is abx-dl already available or is it coming? I took a quick dive and didn't see a repo under the org. I'm happy to help package this up once it is available.

Not currently available, it should be out soon after v0.9 is released.

Currently `mkdir tmp_data && cd tmp_data; archivebox install; archivebox add ...` is effectively equivalent to what `abx-dl` will do.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#136

Post by nikisweeting »

xiconfjs wrote: post #127
wongarsu wrote: post #16

Does this mean it's now possible to write plugins that dismiss cookie popups, solve captchas, scroll web pages etc.?

You mean ArchiveBox still doesn’t deal with cookie popups? If so, it’s quasi not useful for EU based web sites.

It does, you just have to set up a chrome profile that has an extension to hide cookie popups, or use a profile where you've already accepted/closed them and have a session.

You can archive with any chrome profile with arbitrary extensions enabled, so you can use uBlock, I still Don't care about cookies, Ghostery, etc.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#137

Post by nikisweeting »

A4ET8a8uTh0 wrote: post #107
nikisweeting wrote: post #104

Earlier quoted context omitted.

Ah ok. One caveat: it's only available via the 'archivebox shell' / Python API currently, the CLI & web UIs for full depth crawling will come later. You can play around with the models and tasks, but I would wait a few weeks for it to stabilize and check again, it's still under heavy active development Check archivebox/archivebox:dev periodically

No worries. I can do that. You guys probably hear it all the time, but you are doing lords work. If I thought I could be of use in that project, I would be trying to contribute myself ( in fact, let me see if there a way I can participate in a useful manner ).

Thanks! I love working on archiving so far, and it's been very motivating to see more and more people getting into archiving lately.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#138

Post by manofmanysmiles »

nikisweeting wrote: post #130
manofmanysmiles wrote: post #124

Earlier quoted context omitted.

No, definitely not. I have no inherent reason to trust the people working at the Internet Archive over let's say close friend. For me trust is always a human to human concept, and no amount of tech or institutions will change that. The more people I hear making a claim, the more I'm likely to deem the claim(s) as true. This is even true regarding the claims that cryptographic algorithms have the properties that make…

I think the best solution is to have multiple people with reputation attest to the encrypted TLS content without being able to see the cleartext of it, that way they cant easily tamper with it. See my comments on TLSNotary stuff below...

Woah, cool, yes, exactly this!

I think I read a paper or blog post about this concept a while ago, but never saw it implemented!

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#139

Post by ikreymer »

CorentinB wrote: post #129
ikreymer wrote: post #123

Earlier quoted context omitted.

Unfortunately, there is not much we can do about transfer-encoding, but the data is otherwise exactly as is returned from the browser. Browsertrix uses the browser to create web archives, so users get an accurate representation of what they see in their browser, which is generally what people want from archives. We do the best we can with a limited standard that is difficult to modify. Archiving is always lossy, we t…

You could use a proxy. "Archiving is always lossy" No.

Every archiving tool out there makes trade-offs about what is archived and how. No one preserves the raw TLS encrypted H3 traffic because that's not useful. When you browse through an archiving MITM proxy, there are different trade-offs: there's an extra HTTP connection involved (that's not stored), a fake MITM cert, and a downgrade of H2/H3 connection to HTTP/1 (some sites serve different content via H2 vs HTTP/1.1, can detect differences, etc...)

The web is best-effort, and so is archiving the web.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#140

Post by [deleted] »

nikisweeting wrote: post #132
CorentinB wrote: post #128

Earlier quoted context omitted.

He didn't say you modify the data for no reason, he said you violate the standard. Which is true. You could respect it, but you don't.

imo the Webrecorder stuff is truly state of the art, if they're pushing the limits of WARC standards it's for good reason, and I trust their judgement. They pioneered the newer WACZ standard and are really pushing the whole field forward.

[deleted]
Post reply on HN