Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running ArchiveTeam's Warrior in Kubernetes

gabrielsimmer.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Running ArchiveTeam's Warrior in Kubernetes

#21

Post by mplewis »

jfkrrorj wrote: post #11
badlibrarian wrote: post #8

Earlier quoted context omitted.

Suspicion warranted, but citation needed. For now my money is on archives.gov over archive.org. And loc.gov over a website that prioritizes making Pac-Man and Donkey Kong playable in the browser yet leaked the drivers licenses and passports of its patrons, and whose public policy on their wonky javascript UX is "don't read books on a phone."

[flagged]

Hey man, you only created your account one day ago and you're going to town flooding this site with bad takes. What's the deal?

Re: Running ArchiveTeam's Warrior in Kubernetes

#22

Post by badlibrarian »

gumball-amp wrote: post #19
badlibrarian wrote: post #14

Earlier quoted context omitted.

I would classify the end of term web archive (which archive.org is, in its typical fashion, taking far too much credit for) as an example of entities doing things right. https://eotarchive.org/partners/ And saying "archive.org is outside the reach of the US government" -- hell, it's not even outside the reach of the RIAA or the book company with the little penguin on the cover. We should have proper supervised federa…

I'm interested in why you are saying that the Internet Archive is taking too much credit for the end of term web archive. The website you link to demonstrates that it's run by the Internet Archive, although various partners have joined it since it began. Is that not correct? > And I don't know what Archive Team is but maybe they could update their site to provide some information on the people involved. You don't nee…

> Is that not correct?

It's not run by the Archive. It's a collaboration. They didn't even do all the crawling, and the Library of Congress keeps a copy.

https://eotarchive.org/about/

As for Archive Team, their site declares "Archive Team is a loose collective of rogue archivists, programmers, writers and loudmouths."

Dedication is great. And radicalization in response to copyright and preservation certainly deserves some leeway. But a little professionalism wouldn't hurt and the 2600-era roleplay isn't fooling anyone.

Re: Running ArchiveTeam's Warrior in Kubernetes

#23

Post by Tijdreiziger »

badlibrarian wrote: post #8
jfkrrorj wrote: post #7

Earlier quoted context omitted.

[flagged]

Suspicion warranted, but citation needed. For now my money is on archives.gov over archive.org. And loc.gov over a website that prioritizes making Pac-Man and Donkey Kong playable in the browser yet leaked the drivers licenses and passports of its patrons, and whose public policy on their wonky javascript UX is "don't read books on a phone."

> leaked the drivers licenses and passports of its patrons

Source?

Re: Running ArchiveTeam's Warrior in Kubernetes

#24

Post by badlibrarian »

Tijdreiziger wrote: post #23
badlibrarian wrote: post #8

Earlier quoted context omitted.

Suspicion warranted, but citation needed. For now my money is on archives.gov over archive.org. And loc.gov over a website that prioritizes making Pac-Man and Donkey Kong playable in the browser yet leaked the drivers licenses and passports of its patrons, and whose public policy on their wonky javascript UX is "don't read books on a phone."

> leaked the drivers licenses and passports of its patrons Source?

https://www.newsweek.com/catastrophic-internet-archive-hack-...

https://www.newsweek.com/internet-archive-hacked-zendesk-197...

-----

Subject: Notice of Data Security Incident

January 6, 2025

I write on behalf of Internet Archive to inform you about a security incident that involved personal information about you. We regret that this incident occurred and take the security of personal information seriously.

On October 20, 2024, we discovered suspicious activity involving our customer service platform. Specifically, between October 17, 2024 and October 20, 2024, an unauthorized actor obtained access to our customer service platform, which contained information about requests from certain Internet Archive users.

As soon as we learned of the incident, we took action to contain the incident, including by terminating the unauthorized access, and then launched an investigation to determine the nature and scope of the access.

We have determined that the personal information involved in this incident included your name and government ID information such as a driver’s license or passport.

As noted above, we took action to contain the incident and investigate it, including by temporarily taking the customer service platform offline.

You should always remain vigilant for incidents of fraud and identity theft, including by regularly reviewing and monitoring your accounts. If you discover any suspicious or unusual activity on your accounts or suspect identity theft or fraud, be sure to report it immediately to your financial institutions. If you have been a victim of fraud, you can report it to your local police.

Please know that we regret any inconvenience or concern this incident may cause you. Please do not hesitate to contact us at info@archive.org if you have any questions or concerns.

Sincerely, Internet Archive

Re: Running ArchiveTeam's Warrior in Kubernetes

#25

Post by badlibrarian »

toomuchtodo wrote: post #20
badlibrarian wrote: post #18

Earlier quoted context omitted.

If the checks and balances of NARA and LOC (6,000 employees, $1.5 billion in annual funding) is Brewster Kahle asking for $10 on pages serving pirated Nintendo games, then we're in a bit of trouble, aren't we?

On the contrary, the fact that a single person's charitable digital archive can stand toe to toe with a global superpower's archival efforts is a sign that success is possible. We may see things differently though, and that's fine. Would I want to fund NARA and LOC more? Or the Internet Archive? I prefer the latter. Checks and balances. I have donated $10 on your behalf (in addition to my annual donations). (lots of…

I think archive.org deserves more funding and I also think they need to decide if they're an archive, a library, or a pirate site. Since each has a different set of costs, legal risk, and projected longevity.

For the record my opinion is that they need to focus on archival and with a few tweaks could make it safe for more users to upload more material. Going legit archive (as their name implies) instead of hiding behind the DMCA and playing high-stakes poker with copyright law would also make it possible for more entities to provide direct support.

I also disagree that NARA and LoC is subject to whims of the electorate. The Library of Congress is set up to serve, well, Congress. Who funds it. Lotta barriers to cross there, even in these weird times.

I'll take that risk over one guy with limited governance who seems genuinely surprised that he keeps gets hacked and sued. There's a chance the whole thing goes away because he couldn't resist serving up free Frank Sinatra records and got hit with a $621 million lawsuit after he thrice refused to take the stuff down.

Re: Running ArchiveTeam's Warrior in Kubernetes

#26

Post by lopkeny12ko »

NortySpock wrote: post #13
ch71r22 wrote: post #10

For anyone else interested in running this, it only took a couple seconds to launch their docker-compose.yml https://github.com/ArchiveTeam/warrior-dockerfile/blob/maste...

I noticed from the docker overlay filesystem that the container was spraying files all over the disk. (Ephemeral, destroyed on container shutdown, sure, but I wanted to reduce write-wear on my ssd...) I tried setting it up with /tmp as a tmpfs (ramdisk) but it then refused to start... Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?

> the container was spraying files all over the disk

Right, that's basically the point...the Warrior downloads files, compresses them, and uploads them for archival. This necessarily requires staging the files somewhere between download and upload.

> Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?

Why would you want this? This sounds like a terrible footgun.

Re: Running ArchiveTeam's Warrior in Kubernetes

#27

Post by j4ah4n »

NortySpock wrote: post #13
ch71r22 wrote: post #10

For anyone else interested in running this, it only took a couple seconds to launch their docker-compose.yml https://github.com/ArchiveTeam/warrior-dockerfile/blob/maste...

I noticed from the docker overlay filesystem that the container was spraying files all over the disk. (Ephemeral, destroyed on container shutdown, sure, but I wanted to reduce write-wear on my ssd...) I tried setting it up with /tmp as a tmpfs (ramdisk) but it then refused to start... Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?

I think you'll just need to mount it at the right place, with right permissions.

Demonstrated here https://stackoverflow.com/questions/39193419/docker-in-memor...

Re: Running ArchiveTeam's Warrior in Kubernetes

#28

Post by Havoc »

NortySpock wrote: post #13
ch71r22 wrote: post #10

For anyone else interested in running this, it only took a couple seconds to launch their docker-compose.yml https://github.com/ArchiveTeam/warrior-dockerfile/blob/maste...

I noticed from the docker overlay filesystem that the container was spraying files all over the disk. (Ephemeral, destroyed on container shutdown, sure, but I wanted to reduce write-wear on my ssd...) I tried setting it up with /tmp as a tmpfs (ramdisk) but it then refused to start... Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?

You can put the entire docker directory in a ramdisk. Same as you would when trying to move it to a secondary harddisk. Risky though as a reboot would wipe everything

Re: Running ArchiveTeam's Warrior in Kubernetes

#29

Post by myself248 »

lopkeny12ko wrote: post #26
NortySpock wrote: post #13

Earlier quoted context omitted.

I noticed from the docker overlay filesystem that the container was spraying files all over the disk. (Ephemeral, destroyed on container shutdown, sure, but I wanted to reduce write-wear on my ssd...) I tried setting it up with /tmp as a tmpfs (ramdisk) but it then refused to start... Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?

> the container was spraying files all over the disk Right, that's basically the point...the Warrior downloads files, compresses them, and uploads them for archival. This necessarily requires staging the files somewhere between download and upload. > Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container? Why would you want this? This sounds like a terrible footgun.

The Warrior doesn't resume old jobs after a power cycle, so what's the point of committing anything at all to non-volatile storage?

Re: Running ArchiveTeam's Warrior in Kubernetes

#30

Post by rustyminnow »

lopkeny12ko wrote: post #26
NortySpock wrote: post #13

Earlier quoted context omitted.

I noticed from the docker overlay filesystem that the container was spraying files all over the disk. (Ephemeral, destroyed on container shutdown, sure, but I wanted to reduce write-wear on my ssd...) I tried setting it up with /tmp as a tmpfs (ramdisk) but it then refused to start... Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?

> the container was spraying files all over the disk Right, that's basically the point...the Warrior downloads files, compresses them, and uploads them for archival. This necessarily requires staging the files somewhere between download and upload. > Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container? Why would you want this? This sounds like a terrible footgun.

They say exactly why they want it... "I wanted to reduce write-wear on my ssd"
Post reply on HN