Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox: Open-source self-hosted web archive

github.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: ArchiveBox: Open-source self-hosted web archive

#41

Post by aloer »

mysterydip wrote: post #37
samatman wrote: post #36

Earlier quoted context omitted.

I agree with the thrust of this and suggest taking it further by using SQLite for the underlying file format. The advantages of SQLite are too numerous to list, and it has a built-in compressed format so there's no bloat to worry about vs. ZIP files. As an example, this should make it practical, given a few iterations, to store multiple snapshots as deltas, deduplicating identical content. It also obviates having to…

One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.

Remembered I read something about this some time ago, looked it up:

SQLite is a recommended storage format (https://www.sqlite.org/locrsf.html) by the library of congress

> As of this writing (2018-05-29) the only other recommended storage formats for datasets are XML, JSON, and CSV

Re: ArchiveBox: Open-source self-hosted web archive

#42

Post by toomuchtodo »

progval wrote: post #34
toomuchtodo wrote: post #18

Earlier quoted context omitted.

When archiving, you want the request and response headers and the content. Grab-site does that (any tool really that'll write WARC files). Sorry if my comment was ambiguous in that regard. https://en.wikipedia.org/wiki/Web_ARChive https://www.loc.gov/preservation/digital/formats/fdd/fdd0002... If you wanted to write WARC files out with wget, you'd use the options specified in https://www.archiveteam.org/index.php?tit…

Depends on what you want to use your archive for. If it's just for opening it locally in a web browser, mirroring the content is enough.

Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.

Re: ArchiveBox: Open-source self-hosted web archive

#43

Post by greglindahl »

aloer wrote: post #41
mysterydip wrote: post #37

Earlier quoted context omitted.

One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.

Remembered I read something about this some time ago, looked it up: SQLite is a recommended storage format ( https://www.sqlite.org/locrsf.html ) by the library of congress > As of this writing (2018-05-29) the only other recommended storage formats for datasets are XML, JSON, and CSV

The Library of Congress stores its web archives as WARCs.

Re: ArchiveBox: Open-source self-hosted web archive

#44

Post by greglindahl »

burtonator wrote: post #30
peterwwillis wrote: post #28

Earlier quoted context omitted.

Why don't they store everything as plain documents in a zip file and keep metadata in a json? Seems more future proof/easier for users to manipulate than WARC

This is what Polar does but WARC is standardized... I think the argument against WARC though is that there isn't really much interchange. the point of standards is interchange IMO

There actually is interchange of WARCs in the web archiving community.

Re: ArchiveBox: Open-source self-hosted web archive

#45

Post by [deleted] »

bnewbold wrote: post #38
peterwwillis wrote: post #28

Earlier quoted context omitted.

Why don't they store everything as plain documents in a zip file and keep metadata in a json? Seems more future proof/easier for users to manipulate than WARC

Have you looked at the WARC format? It's ridiculously simple, basically concatenated raw HTTP requests and responses, with some extra HTTP metadata headers mixed in (a la extra JSON metadata keys). You can open it with a text editor. Very simple and efficient to manipulate, and very efficient to iterate over or generate. https://iipc.github.io/warc-specifications/specifications/wa... Arguably the biggest problem is t…

[deleted]

Re: ArchiveBox: Open-source self-hosted web archive

#46

Post by rakoo »

mysterydip wrote: post #37
samatman wrote: post #36

Earlier quoted context omitted.

I agree with the thrust of this and suggest taking it further by using SQLite for the underlying file format. The advantages of SQLite are too numerous to list, and it has a built-in compressed format so there's no bloat to worry about vs. ZIP files. As an example, this should make it practical, given a few iterations, to store multiple snapshots as deltas, deduplicating identical content. It also obviates having to…

One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.

Very true, but I'd say SQlite is the one application dependency that could have a pass:

- they promise support until at least 2050 (https://sqlite.org/lts.html)

- if a promise isn't enough, SQlite is to be supported for the entire lifetime of the Airbus A350 airframe (https://mobile.twitter.com/copiousfreetime/status/6758345433...). I would assume Airbus will be paying for it.

A real problem with application dependency is that it makes it extremely hard to create a completely independent alternative. A good protocol should have multiple implementations with nothing in common otherwise you rely on specific implementation details. The infamous case of WebSQL should remind us of that.

Re: ArchiveBox: Open-source self-hosted web archive

#47

Post by EamonnMR »

aloer wrote: post #39

I have spent the last hours reading up on everything-WARC that I could find but I still haven't been able to answer my main question: why only as external crawlers? There does not seem to be a tool to actually capture a warc directly in your own browser session. webrecorder ( http://webrecorder.io/ ) is the only example I could find that comes close in terms of user experience but it still requires a third party and…

Definitely report back if you find anything, I would be very interested.

Re: ArchiveBox: Open-source self-hosted web archive

#48

Post by peterwwillis »

bnewbold wrote: post #38
peterwwillis wrote: post #28

Earlier quoted context omitted.

Why don't they store everything as plain documents in a zip file and keep metadata in a json? Seems more future proof/easier for users to manipulate than WARC

Have you looked at the WARC format? It's ridiculously simple, basically concatenated raw HTTP requests and responses, with some extra HTTP metadata headers mixed in (a la extra JSON metadata keys). You can open it with a text editor. Very simple and efficient to manipulate, and very efficient to iterate over or generate. https://iipc.github.io/warc-specifications/specifications/wa... Arguably the biggest problem is t…

Yeah, the fact that I have to convert them after is just an unnecessary extra step for me, so I'll stick to wget and httrack for my archives. Once I mirror them I can just copy the files anywhere and browse them on any browser/device.

Re: ArchiveBox: Open-source self-hosted web archive

#49

Post by machawk1 »

aloer wrote: post #39

I have spent the last hours reading up on everything-WARC that I could find but I still haven't been able to answer my main question: why only as external crawlers? There does not seem to be a tool to actually capture a warc directly in your own browser session. webrecorder ( http://webrecorder.io/ ) is the only example I could find that comes close in terms of user experience but it still requires a third party and…

When I initially coded up WARCreate, the webRequest API was still experimental. I believe there are more mature APIs that can be used from the extension context but some require DevTools to be visually open, which is not a common usage pattern of a typical web user.

Per the ticket, we have worked on a few other JavaScript-driven web preservation project like https://github.com/N0taN3rd/node-warc and https://github.com/N0taN3rd/Squidwarc, among others.

Web archiving from a browser extension is difficult but can be improved. I don't know of any other approaches at trying to do this via a browser extension beyond submitting a URI to an archive.

Re: ArchiveBox: Open-source self-hosted web archive

#50

Post by mysterydip »

aloer wrote: post #41
mysterydip wrote: post #37

Earlier quoted context omitted.

One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.

Remembered I read something about this some time ago, looked it up: SQLite is a recommended storage format ( https://www.sqlite.org/locrsf.html ) by the library of congress > As of this writing (2018-05-29) the only other recommended storage formats for datasets are XML, JSON, and CSV

Good to know, thanks!
Post reply on HN