Live data from Hacker News

Running ArchiveTeam's Warrior in Kubernetes

gabrielsimmer.com

11–20 of 40 posts

Re: Running ArchiveTeam's Warrior in Kubernetes

#11
post #7

Earlier quoted context omitted.

[flagged]

Suspicion warranted, but citation needed. For now my money is on archives.gov over archive.org. And loc.gov over a website that prioritizes making Pac-Man and Donkey Kong playable in the browser yet leaked the drivers licenses and passports of its patrons, and whose public policy on their wonky javascript UX is "don't read books on a phone."

[flagged]

Re: Running ArchiveTeam's Warrior in Kubernetes

#12

Earlier quoted context omitted.

> by proper entities as required by federal law. What federal law do you suppose is guiding the mass deletions? That doesn't look like archiving to me. Now that the foxes are running the henhouse, how reliable do you suppose their own archives are?

Some of the mass deletions are merely a new administration setting up shop. Policies from the previous administration don't belong on the current whitehouse.gov. They wind up here instead https://bidenwhitehouse.archives.gov/ We pay half a billion in tax dollars for the National Archives, and nearly a billion to the Library of Congress to preserve these records. Others are managed as part of Presidential Libraries. T…

> They let vagrants sleep on their steps yet seem surprised when they set the utility pole outsides on fire.

Tbf I have let many people sleep on my doorstep and none of them tried to set my building on fire. One of them even sang for me; he had a killer baritone. Overall it seems like a fairly harmless thing.

Re: Running ArchiveTeam's Warrior in Kubernetes

#13
post #10

For anyone else interested in running this, it only took a couple seconds to launch their docker-compose.yml https://github.com/ArchiveTeam/warrior-dockerfile/blob/maste...

I noticed from the docker overlay filesystem that the container was spraying files all over the disk. (Ephemeral, destroyed on container shutdown, sure, but I wanted to reduce write-wear on my ssd...)

I tried setting it up with /tmp as a tmpfs (ramdisk) but it then refused to start...

Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?

Re: Running ArchiveTeam's Warrior in Kubernetes

#14

Earlier quoted context omitted.

Some of the mass deletions are merely a new administration setting up shop. Policies from the previous administration don't belong on the current whitehouse.gov. They wind up here instead https://bidenwhitehouse.archives.gov/ We pay half a billion in tax dollars for the National Archives, and nearly a billion to the Library of Congress to preserve these records. Others are managed as part of Presidential Libraries. T…

Archive.org is outside of the reach of the US government, and is globally distributed. When the US government deletes or darks data (as it has recently done across wide swaths of the federal government website properties), you have no recourse. This means your argument about the resources that go into the US government as a data custodian are meaningless: the outcome is what is material, which is the archival and lon…

I would classify the end of term web archive (which archive.org is, in its typical fashion, taking far too much credit for) as an example of entities doing things right.

https://eotarchive.org/partners/

And saying "archive.org is outside the reach of the US government" -- hell, it's not even outside the reach of the RIAA or the book company with the little penguin on the cover.

We should have proper supervised federal archiving and archive.org should be far better run, too.

And I don't know what Archive Team is but maybe they could update their site to provide some information on the people involved. And perhaps update their understanding of what's possible with docker containers while they're at it.

Because the counterpoint to a radicalized Musk screwing around with government databases isn't an opposing group of anonymous radicals screwing around with commercial databases.

Re: Running ArchiveTeam's Warrior in Kubernetes

#15

Earlier quoted context omitted.

Some of the mass deletions are merely a new administration setting up shop. Policies from the previous administration don't belong on the current whitehouse.gov. They wind up here instead https://bidenwhitehouse.archives.gov/ We pay half a billion in tax dollars for the National Archives, and nearly a billion to the Library of Congress to preserve these records. Others are managed as part of Presidential Libraries. T…

> They let vagrants sleep on their steps yet seem surprised when they set the utility pole outsides on fire. Tbf I have let many people sleep on my doorstep and none of them tried to set my building on fire. One of them even sang for me; he had a killer baritone. Overall it seems like a fairly harmless thing.

I wasn't speaking metaphorically. Fire set to pole. Site went down.

Re: Running ArchiveTeam's Warrior in Kubernetes

#16

Earlier quoted context omitted.

Archive.org is outside of the reach of the US government, and is globally distributed. When the US government deletes or darks data (as it has recently done across wide swaths of the federal government website properties), you have no recourse. This means your argument about the resources that go into the US government as a data custodian are meaningless: the outcome is what is material, which is the archival and lon…

I would classify the end of term web archive (which archive.org is, in its typical fashion, taking far too much credit for) as an example of entities doing things right. https://eotarchive.org/partners/ And saying "archive.org is outside the reach of the US government" -- hell, it's not even outside the reach of the RIAA or the book company with the little penguin on the cover. We should have proper supervised federa…

Agree the US government should contribute in some capacity. Agree they should be robustly funded to do this. But, checks and balances are also important, and when a node goes rouge or dark, the system must be fault tolerant and operate when degradation occurs. I previously said "I trust Brewster and the rest of the IA gang more than the US government to safeguard the Internet Archive." [1] I feel this assertion has been proven out over the last few weeks.

ArchiveTeam stands on its own as an independent, community driven volunteer digital archival and preservation effort. If you don't understand why, what, and how they operate, look closer and be more curious [2].

[1] https://news.ycombinator.com/item?id=41984664

[2] https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence

Re: Running ArchiveTeam's Warrior in Kubernetes

#18

Earlier quoted context omitted.

I would classify the end of term web archive (which archive.org is, in its typical fashion, taking far too much credit for) as an example of entities doing things right. https://eotarchive.org/partners/ And saying "archive.org is outside the reach of the US government" -- hell, it's not even outside the reach of the RIAA or the book company with the little penguin on the cover. We should have proper supervised federa…

Agree the US government should contribute in some capacity. Agree they should be robustly funded to do this. But, checks and balances are also important, and when a node goes rouge or dark, the system must be fault tolerant and operate when degradation occurs. I previously said "I trust Brewster and the rest of the IA gang more than the US government to safeguard the Internet Archive." [1] I feel this assertion has b…

If the checks and balances of NARA and LOC (6,000 employees, $1.5 billion in annual funding) is Brewster Kahle asking for $10 on pages serving pirated Nintendo games, then we're in a bit of trouble, aren't we?

Re: Running ArchiveTeam's Warrior in Kubernetes

#19

Earlier quoted context omitted.

Archive.org is outside of the reach of the US government, and is globally distributed. When the US government deletes or darks data (as it has recently done across wide swaths of the federal government website properties), you have no recourse. This means your argument about the resources that go into the US government as a data custodian are meaningless: the outcome is what is material, which is the archival and lon…

I would classify the end of term web archive (which archive.org is, in its typical fashion, taking far too much credit for) as an example of entities doing things right. https://eotarchive.org/partners/ And saying "archive.org is outside the reach of the US government" -- hell, it's not even outside the reach of the RIAA or the book company with the little penguin on the cover. We should have proper supervised federa…

I'm interested in why you are saying that the Internet Archive is taking too much credit for the end of term web archive. The website you link to demonstrates that it's run by the Internet Archive, although various partners have joined it since it began.

Is that not correct?

> And I don't know what Archive Team is but maybe they could update their site to provide some information on the people involved.

You don't need to reveal your identity, but looking through your comments, it looks like you originally spun up this account to criticize the Internet Archive. I'll just note that accusing others of being "anonymous radicals" falls a little flatter when you're anonymous yourself.

(Relevant disclosure: I've worked with IA and Brewster Kahle, and defended him here before.)

Re: Running ArchiveTeam's Warrior in Kubernetes

#20

Earlier quoted context omitted.

Agree the US government should contribute in some capacity. Agree they should be robustly funded to do this. But, checks and balances are also important, and when a node goes rouge or dark, the system must be fault tolerant and operate when degradation occurs. I previously said "I trust Brewster and the rest of the IA gang more than the US government to safeguard the Internet Archive." [1] I feel this assertion has b…

If the checks and balances of NARA and LOC (6,000 employees, $1.5 billion in annual funding) is Brewster Kahle asking for $10 on pages serving pirated Nintendo games, then we're in a bit of trouble, aren't we?

On the contrary, the fact that a single person's charitable digital archive can stand toe to toe with a global superpower's archival efforts is a sign that success is possible. We may see things differently though, and that's fine. Would I want to fund NARA and LOC more? Or the Internet Archive? I prefer the latter. Checks and balances. I have donated $10 on your behalf (in addition to my annual donations).

(lots of good people at NARA and the LOC, but they are subject to the whims of the US electorate, which is not great; the Internet Archive is not)

Post reply on HN