Live data from Hacker News

Help preserve the internet with Archiveteam's warrior

selfhostedheaven.com

51–53 of 53 posts

Re: Help preserve the internet with Archiveteam's warrior

#51

Yes! Archiving is important, we have already seen so much online history gone down the drain or just accidentally saved. Large institutions like the internet archive are doing an admirable job, but there is a lot of content that they cannot and will not cover. So we will definitely (also) need volunteer-based archival for the foreseeable future. 18TB drives are ~$300 a piece right now, go buy one and help our collect…

ArchiveTeam sends archives to Internet Archive but the two are not related. I don't think you confused the two but I mention this every time just in case. The Warrior is a small Docker image that downloads files via your ISP connection and forwards them to the AT servers. No need for large drives. For my personal use, I have a home server install of https://github.com/ArchiveBox/ArchiveBox and for that one you may wa…

The last time I tried to do anything with or for the Archive Team, it was a mostly "just watch us work" sort of deal.

The tools couldn't be built without additional knowledge that wasn't published anywhere -- because there had been drift from what was published versus what was working, and those changes never got folded back in. And there were multiple versions and variants of the tools, with different teams using different versions or variants.

And once you built the tools, you couldn't get your Warrior into the list to be used, although you could always run your systems separately.

It's not like you could sign up for a SETI@Home type initiative and just let your equipment run.

I understand why they work this way. It's a very insular crowd, and new people and resources seem to disappear as quickly as they showed up.

So, they let you watch.

If you stay around long enough (months? years?), then they might let you start participating. But I wasn't willing to wait that long.

Re: Help preserve the internet with Archiveteam's warrior

#52
post #48

Earlier quoted context omitted.

>but my machine crashed Maybe it's not the servers who throttle you then ;)

Wget will exhaust all available ram on a long enough crawl.

No, i crawled many multi-gigabyte sites with my raspberry2 for days.

Re: Help preserve the internet with Archiveteam's warrior

#53
post #52

Earlier quoted context omitted.

Wget will exhaust all available ram on a long enough crawl.

No, i crawled many multi-gigabyte sites with my raspberry2 for days.

I've had memory exhaustion (on a 4GB system) after I think about 600GB in a single crawl. Splitting it into multiple crawls is of course better.

That was a site specifically set up to deal with large collections of files though.

Post reply on HN