Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Help preserve the internet with Archiveteam's warrior

selfhostedheaven.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Help preserve the internet with Archiveteam's warrior

#11

Post by smarx007 »

iforgotpassword wrote: post #8

How likely is it you end up downloading child porn on behalf of them? In other words, how well curated or specific is the list of download jobs your node gets assigned? If it's something like "just grab everything from this blog platform" I guess chances are not zero.

My experience has shown that list to be extremely well-curated. See https://wiki.archiveteam.org/#Warrior-based_projects for the current list.

Though if you join the Reddit archival project, all bets may be off but that's not AT team's fault, I guess.

Re: Help preserve the internet with Archiveteam's warrior

#12

Post by smarx007 »

uniqueuid wrote: post #2

Yes! Archiving is important, we have already seen so much online history gone down the drain or just accidentally saved. Large institutions like the internet archive are doing an admirable job, but there is a lot of content that they cannot and will not cover. So we will definitely (also) need volunteer-based archival for the foreseeable future. 18TB drives are ~$300 a piece right now, go buy one and help our collect…

ArchiveTeam sends archives to Internet Archive but the two are not related. I don't think you confused the two but I mention this every time just in case.

The Warrior is a small Docker image that downloads files via your ISP connection and forwards them to the AT servers. No need for large drives.

For my personal use, I have a home server install of https://github.com/ArchiveBox/ArchiveBox and for that one you may want to get some storage, though I prefer to host its data on the SSD for performance reasons (my archive grows approx. 5000 items or 150GB per year). It's like a private Internet Archive on your home network.

Re: Help preserve the internet with Archiveteam's warrior

#13

Post by charcircuit »

mhitza wrote: post #10
iforgotpassword wrote: post #8

How likely is it you end up downloading child porn on behalf of them? In other words, how well curated or specific is the list of download jobs your node gets assigned? If it's something like "just grab everything from this blog platform" I guess chances are not zero.

I think you would be more likely to win the lottery without playing. That type of content has long moved from clearnet to the darknet. I would be inexplicably surprised if that type of content can be found on the clearnet. But I still can be wrong. However if you're in the US loli hentai is going to be a risk and legal headache for sure https://www.shouselaw.com/ca/blog/is-loli-illegal-in-the-uni... As far as I'm awa…

>That type of content has long moved from clearnet to the darknet.

A fraction of it.

>I would be inexplicably surprised if that type of content can be found on the clearnet

That kind of content is a single internet search away.

Re: Help preserve the internet with Archiveteam's warrior

#15

Post by capableweb »

RNAlfons wrote: post #14

Make it an easy installable/runable Windows application and it will spread like wildfire.

If it was only that easy. To make distributed archiving as high quality as possible, you need reproducible environments as much as possible, which is why the "official" way of participating is to run virtual machines, instead of directly on the host.

Not sure why this 3rd party is the submission site rather than the official page, which is this: https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior

Has a couple of different installation methods as well.

Re: Help preserve the internet with Archiveteam's warrior

#16

Post by uniqueuid »

smarx007 wrote: post #12
uniqueuid wrote: post #2

Yes! Archiving is important, we have already seen so much online history gone down the drain or just accidentally saved. Large institutions like the internet archive are doing an admirable job, but there is a lot of content that they cannot and will not cover. So we will definitely (also) need volunteer-based archival for the foreseeable future. 18TB drives are ~$300 a piece right now, go buy one and help our collect…

ArchiveTeam sends archives to Internet Archive but the two are not related. I don't think you confused the two but I mention this every time just in case. The Warrior is a small Docker image that downloads files via your ISP connection and forwards them to the AT servers. No need for large drives. For my personal use, I have a home server install of https://github.com/ArchiveBox/ArchiveBox and for that one you may wa…

Thanks, it's always good to point that out.

There's a surprising amount of tools that are able to submit data to the internet archive (and get data from there). Even wget can produce WARC archive files.

While the warrior downloads content via your line (a bit like a residential proxy network), I do think it's important that we decentralize the storage as well.

Just without the crypto mafia/drug traders/investors.

Re: Help preserve the internet with Archiveteam's warrior

#17

Post by uniqueuid »

Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

Just to point this out, on a technical level, the internet archive has very (!) little overhead.

Crawled data is de-duplicated on the request level and response payloads can be individually gzipped as well as having per-archive-file compression. [1]

[1] https://www.iso.org/standard/68004.html

Re: Help preserve the internet with Archiveteam's warrior

#18

Post by uniqueuid »

prox wrote: post #4
uniqueuid wrote: post #2

Yes! Archiving is important, we have already seen so much online history gone down the drain or just accidentally saved. Large institutions like the internet archive are doing an admirable job, but there is a lot of content that they cannot and will not cover. So we will definitely (also) need volunteer-based archival for the foreseeable future. 18TB drives are ~$300 a piece right now, go buy one and help our collect…

I kind of wonder how we can make it searchable again. Is this included in this archiving effort? In any case wonderful work.

There is a standard set of tooling for indexing archives: CDX files. [1]

They index WARC archives and can be used to quickly find records. You can build on top of this (and some systems do) to make a proper search front-end.

But in general, these archives are NOT geared towards full-blown search because it would be pretty expensive to keep the indexes in hot cache. Plus you would need to deal with historical versions of records, which is not normally done in search UX.

[1] https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem#CD...

Re: Help preserve the internet with Archiveteam's warrior

#19

Post by janandonly »

Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

Unlike the Archive, the "value" of Bitcoin can be measured: Today's market cap of BTC is $839.5B

Re: Help preserve the internet with Archiveteam's warrior

#20

Post by londons_explore »

DoingIsLearning wrote: post #7
Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

I disagree, the unfiltered high noise is what makes it valuable. Curation is a bias. If someone wants to dive into any topic in the archive 30 years from now they will have access to everything, not access to what some of us deem 'worthy' of curating. I agree that it makes it harder to find things but I also see the value of IA as a time capsule.

Tools to separate signal from noise will also get better in the future. You can imagine that in 100 years time, using a super duper AI search engine will perform far better than whatever some human decided to categorize stuff as today.
Post reply on HN