Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Help preserve the internet with Archiveteam's warrior

selfhostedheaven.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Help preserve the internet with Archiveteam's warrior

#21

Post by zdkl »

uniqueuid wrote: post #16
smarx007 wrote: post #12

Earlier quoted context omitted.

ArchiveTeam sends archives to Internet Archive but the two are not related. I don't think you confused the two but I mention this every time just in case. The Warrior is a small Docker image that downloads files via your ISP connection and forwards them to the AT servers. No need for large drives. For my personal use, I have a home server install of https://github.com/ArchiveBox/ArchiveBox and for that one you may wa…

Thanks, it's always good to point that out. There's a surprising amount of tools that are able to submit data to the internet archive (and get data from there). Even wget can produce WARC archive files. While the warrior downloads content via your line (a bit like a residential proxy network), I do think it's important that we decentralize the storage as well. Just without the crypto mafia/drug traders/investors.

AFAIK you can use IPFS (& clusters[0]) without relying on the crypto parts of that ecosystem. That ought to fit rather well with the use case.

[0] https://cluster.ipfs.io/

Re: Help preserve the internet with Archiveteam's warrior

#22

Post by cyber_kinetist »

Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

I think the real problem is a bit deeper: Unorganized raw data itself is of very low value, but it becomes much more valuable when humans process, categorize, and interpret it via a higher-level system of reason. We're doing a lot of the former but not the latter: we have so much data but have no idea what they all mean as a whole.

Libraries aren't just "a bunch of books piled up in shelves", they're a historical invention built and perfected for centuries where books are extensively coded and catalogued via a complex hierarchical system. As we are dealing with far more data than the past (not just books but posts and comments from all over the world, as well as new kinds of media such as images and videos), and also have new kinds of conceptual and technological inventions that previous librarians didn't have access to (hyperlinks, databases, graph theory, machine learning, etc.), the current status of data management begs for a major overhaul. (For example, the best we are currently doing for querying and searching from massive data is Google, and it is incredibly primitive! And even then we lament that the quality of it has decreased in favor of SEO-maximizing content.) So much raw data is created every day, and we just seem to fail to understand and interpret almost all of it, I see it as one of the major historical crises we face today. Instead of just storing data, we must find radical new methodologies and tools to search, filter, and explore data, and this poses as both a philosophical problem (of semiotics, linguistics, and hermeneutics) as well as a technological problem.

Re: Help preserve the internet with Archiveteam's warrior

#23

Post by uniqueuid »

zdkl wrote: post #21
uniqueuid wrote: post #16

Earlier quoted context omitted.

Thanks, it's always good to point that out. There's a surprising amount of tools that are able to submit data to the internet archive (and get data from there). Even wget can produce WARC archive files. While the warrior downloads content via your line (a bit like a residential proxy network), I do think it's important that we decentralize the storage as well. Just without the crypto mafia/drug traders/investors.

AFAIK you can use IPFS (& clusters[0]) without relying on the crypto parts of that ecosystem. That ought to fit rather well with the use case. [0] https://cluster.ipfs.io/

Yes there are some really interesting projects, also in the ML replicability space.

One really nice approach is the DAT project [1]. The protocol [2] looks pretty sensible and useful. Unfortunately, the tooling has been in such a state of permanent flux (i.e. perpetual deprecation) that I've never bothered to invest much time.

[1] https://datproject.org/

[1] https://datproject.org/

Re: Help preserve the internet with Archiveteam's warrior

#25

Post by cxr »

RNAlfons wrote: post #14

Make it an easy installable/runable Windows application and it will spread like wildfire.

Why even require that? If the data in question is available over HTTP, it should be as easy as opening a page from the relevant origin in a browser tab, optionally opening a second tab for a "Warrior Dashboard", then invoking a bookmarklet on the former to slurp up data by XHR &tc. (If it's necessary to cross origins as the thing roves around, the dashboard can alert you to this while it continues doing what it can with the first origin. Just have the human return to the dashboard from time to time and repeat the second step to run as many in parallel as they want.)

Re: Help preserve the internet with Archiveteam's warrior

#26

Post by jrwr »

capableweb wrote: post #15
RNAlfons wrote: post #14

Make it an easy installable/runable Windows application and it will spread like wildfire.

If it was only that easy. To make distributed archiving as high quality as possible, you need reproducible environments as much as possible, which is why the "official" way of participating is to run virtual machines, instead of directly on the host. Not sure why this 3rd party is the submission site rather than the official page, which is this: https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior Has a couple…

Yep, Using Virtual box is rather easy to get the warrior running!

Re: Help preserve the internet with Archiveteam's warrior

#27

Post by camtarn »

uniqueuid wrote: post #18
prox wrote: post #4

Earlier quoted context omitted.

I kind of wonder how we can make it searchable again. Is this included in this archiving effort? In any case wonderful work.

There is a standard set of tooling for indexing archives: CDX files. [1] They index WARC archives and can be used to quickly find records. You can build on top of this (and some systems do) to make a proper search front-end. But in general, these archives are NOT geared towards full-blown search because it would be pretty expensive to keep the indexes in hot cache. Plus you would need to deal with historical versions…

Ah, is the WARC format the reason it's called 'Warrior'? It seems like a very strange name for an archival program.

Re: Help preserve the internet with Archiveteam's warrior

#29

Post by lkbm »

DoingIsLearning wrote: post #7
Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

I disagree, the unfiltered high noise is what makes it valuable. Curation is a bias. If someone wants to dive into any topic in the archive 30 years from now they will have access to everything, not access to what some of us deem 'worthy' of curating. I agree that it makes it harder to find things but I also see the value of IA as a time capsule.

Yes, curation is very valuable, but it needs to be a layer on top of an uncurated source.

I enjoy using Open Library to re-read obscure middle grades books from the 1950s-1990, and there are some obscure DOS games I want to revisit. It's hard to find what I want sometimes, but only having access to curated lists would change it from "hard" to "impossible" in many cases.

Re: Help preserve the internet with Archiveteam's warrior

#30

Post by qiskit »

Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

> tremendous energy and amounts of resources are being put towards very little of value.

I doubt it take tremendous energy or resources. What percentage of the overall internet energy/resources is used by IA? An insignificant minuscule amount.

The problem with IA is that they are constantly attacked by institutions, corporations, etc to remove content.

Post reply on HN