Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running ArchiveTeam's Warrior in Kubernetes

gabrielsimmer.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Running ArchiveTeam's Warrior in Kubernetes

#31

Post by michaelt »

NortySpock wrote: post #13
ch71r22 wrote: post #10

For anyone else interested in running this, it only took a couple seconds to launch their docker-compose.yml https://github.com/ArchiveTeam/warrior-dockerfile/blob/maste...

I noticed from the docker overlay filesystem that the container was spraying files all over the disk. (Ephemeral, destroyed on container shutdown, sure, but I wanted to reduce write-wear on my ssd...) I tried setting it up with /tmp as a tmpfs (ramdisk) but it then refused to start... Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?

> I wanted to reduce write-wear on my ssd

Modern SSDs are pretty good at things like wear levelling.

For example [1] reports that a bunch of 256 GB SSDs lasted to 2000+ terabytes written, and a handful up to 7000 terabytes written. So you could saturate a 100 megabit internet connection for 5 years before even a small SSD would wear out. And an SSD 4x the size has 4x the life.

If you're running on a raspberry pi with a microsd card for storage, feel free to keep worrying though :)

[1] https://www.reddit.com/r/chia/comments/mukiwz/are_we_overthi...

Re: Running ArchiveTeam's Warrior in Kubernetes

#32

Post by johnklos »

badlibrarian wrote: post #5
myself248 wrote: post #3

Earlier quoted context omitted.

> by proper entities as required by federal law. What federal law do you suppose is guiding the mass deletions? That doesn't look like archiving to me. Now that the foxes are running the henhouse, how reliable do you suppose their own archives are?

Some of the mass deletions are merely a new administration setting up shop. Policies from the previous administration don't belong on the current whitehouse.gov. They wind up here instead https://bidenwhitehouse.archives.gov/ We pay half a billion in tax dollars for the National Archives, and nearly a billion to the Library of Congress to preserve these records. Others are managed as part of Presidential Libraries. T…

You really do live up to your name!

You imply that archive.org is somehow doing something wrong by letting "vagrants" sleep on their steps. I'd assert that people who are compassionate are more trustworthy than people who think punishing others should be normalized. I'd definitely prefer my backups in the hands of compassionate people.

The problem is that the people who want to see others be punished can't be trusted to, you know, not do that. Removing information about climate change, about vaccines, about trans care, et cetera, very well could happen at the hands of those who get off on punishing others.

You say the National Archives already does this. What happens when the current administration fires everyone and replaces them with non-professionals?

So I really don't know why you'd be in here talking ish about ArchiveTeam.

Re: Running ArchiveTeam's Warrior in Kubernetes

#33

Post by badlibrarian »

johnklos wrote: post #32
badlibrarian wrote: post #5

Earlier quoted context omitted.

Some of the mass deletions are merely a new administration setting up shop. Policies from the previous administration don't belong on the current whitehouse.gov. They wind up here instead https://bidenwhitehouse.archives.gov/ We pay half a billion in tax dollars for the National Archives, and nearly a billion to the Library of Congress to preserve these records. Others are managed as part of Presidential Libraries. T…

You really do live up to your name! You imply that archive.org is somehow doing something wrong by letting "vagrants" sleep on their steps. I'd assert that people who are compassionate are more trustworthy than people who think punishing others should be normalized. I'd definitely prefer my backups in the hands of compassionate people. The problem is that the people who want to see others be punished can't be trusted…

> I'd definitely prefer my backups in the hands of compassionate people.

I prefer them in the hands of competent people, in a building with climate control.

Heard about the time these compassionate folks tried to run a bank and got shut down in the Obama era?

> Unwillingness to open accounts within the field of membership, make loans, and establish operations in the low-income community where the credit union was chartered to serve

https://ncua.gov/newsroom/press-release/2016/internet-archiv...

Re: Running ArchiveTeam's Warrior in Kubernetes

#35

Post by WildGreenLeave »

The first thing I setup when I started to manage my own Kubernetes cluster more then a year ago was this Warrior, I completely forgot about it until this post.

Has been active for over a year steadily working the recommended project. Downloaded over 3TB in 6 days (node reboot, so pod was restarted and stats are not persistent). So rough extrapolation is about 180TB. Happy to help the good cause of the ArchiveTeam!

Edit: typo

Re: Running ArchiveTeam's Warrior in Kubernetes

#36

Post by crtasm »

ch71r22 wrote: post #10

For anyone else interested in running this, it only took a couple seconds to launch their docker-compose.yml https://github.com/ArchiveTeam/warrior-dockerfile/blob/maste...

How should I approach looking at what that will install before I run it? Every path on the site returns 'nope' https://atdr.meo.ws/archiveteam/warrior-dockerfile

Re: Running ArchiveTeam's Warrior in Kubernetes

#38

Post by toomuchtodo »

badlibrarian wrote: post #25
toomuchtodo wrote: post #20

Earlier quoted context omitted.

On the contrary, the fact that a single person's charitable digital archive can stand toe to toe with a global superpower's archival efforts is a sign that success is possible. We may see things differently though, and that's fine. Would I want to fund NARA and LOC more? Or the Internet Archive? I prefer the latter. Checks and balances. I have donated $10 on your behalf (in addition to my annual donations). (lots of…

I think archive.org deserves more funding and I also think they need to decide if they're an archive, a library, or a pirate site. Since each has a different set of costs, legal risk, and projected longevity. For the record my opinion is that they need to focus on archival and with a few tweaks could make it safe for more users to upload more material. Going legit archive (as their name implies) instead of hiding beh…

> I also disagree that NARA and LoC is subject to whims of the electorate.

National Archives Workers Unsure If Marco Rubio Has Secretly Been Their Boss for Weeks - https://www.404media.co/national-archives-workers-unsure-if-... - February 6th, 2025

Re: Running ArchiveTeam's Warrior in Kubernetes

#39

Post by toomuchtodo »

toomuchtodo wrote: post #38
badlibrarian wrote: post #25

Earlier quoted context omitted.

I think archive.org deserves more funding and I also think they need to decide if they're an archive, a library, or a pirate site. Since each has a different set of costs, legal risk, and projected longevity. For the record my opinion is that they need to focus on archival and with a few tweaks could make it safe for more users to upload more material. Going legit archive (as their name implies) instead of hiding beh…

> I also disagree that NARA and LoC is subject to whims of the electorate. National Archives Workers Unsure If Marco Rubio Has Secretly Been Their Boss for Weeks - https://www.404media.co/national-archives-workers-unsure-if-... - February 6th, 2025

Trump dismisses head of the National Archives - https://www.nbcnews.com/politics/donald-trump/trump-dismisse... - February 7th, 2025

Re: Running ArchiveTeam's Warrior in Kubernetes

#40

Post by badlibrarian »

toomuchtodo wrote: post #39
toomuchtodo wrote: post #38

Earlier quoted context omitted.

> I also disagree that NARA and LoC is subject to whims of the electorate. National Archives Workers Unsure If Marco Rubio Has Secretly Been Their Boss for Weeks - https://www.404media.co/national-archives-workers-unsure-if-... - February 6th, 2025

Trump dismisses head of the National Archives - https://www.nbcnews.com/politics/donald-trump/trump-dismisse... - February 7th, 2025

I do not share your pessimism. And you do not seem to understand the nuances of a representative democracy as it pertains to the Executive branch.

Please report back when the transition from the 11th Archivist to the 12th Archivist causes any data or paper loss.

https://www.archives.gov/about/history/archivists

I trust these folks far more than the "not affiliated with archive.org" Archive Team and their wget scripts that somehow jam data via backdoor into the web.archive.org database.

https://www.archives.gov/about/organization/senior-staff

If someone pulls nonsense at archives.gov, whistles will be blown and the press will respond. When nonsense goes on at archive.org, I see hagiographies, third party apologists, and people who lack the qualifications to get a job filing paper in a University library mismanaging simple archival projects.

Post reply on HN