Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to become a pirate archivist

annas-blog.org

Post reply on HN
Print view View original

41–50 of 103 posts

Re: How to become a pirate archivist

#41

Post by justshowpost »

I just dropped here to praise archivists and their merit in general. I treasure content (regardless of its perceived quality) preservation much more than legal or even ethical problems associated with it.

Anecdote: Remember when Microsoft Corp. declared what they love open source software and launched CodePlex platform, and then lost their business interest in it (when they bought GitHub) so they completely erased CodePlex archives? I was able to reach several long forgotten project I was interested in thanks to invaluable work of independent volunteer archivists. (It was quite tough manual job for me, I had to d/l database then locate desired archive segment and only then could transfer required files via bittorrent proto)

Re: How to become a pirate archivist

#42

Post by chhs »

londons_explore wrote: post #38
imhoguy wrote: post #36

As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, ev…

The "v2" torrent file format allows most of this. Some clients have support already. All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere. If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.

A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm.

But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there's no lookup mechanism for a "pieces root" to know that torrent B exists and contains file C.

If you were to make a v2 torrent of your entire drive, other clients won't know to download from your torrent. They'd need to have the contents of the metadata to know it would contain a file they are interested in, and have no way of knowing which metadata contains the desired "pieces root" entries without downloading all of them.

I'm very interested in this problem space, if you are aware of clients/mechanisms that allow for this I would love to hear them.

Re: How to become a pirate archivist

#43

Post by teddyh »

est wrote: post #13
ynno wrote: post #3

I think Alexandra Elbakyan actually did not want to be revealed as the librarian behind Sci-Hub, it was her poor opsec that led to her being identified. Basically her servers were set up to emit detailed error messages from PHP, including full path of faulting source file, which was under directory /home/ringo-ring, which could be traced to a username she had online on an unrelated site, attached to her real name. Be…

> which was under directory /home/ringo-ring, which could be traced to a username Ha, my home dir is always called "me" or "and". Try google that.

IIRC, on new installations of NeXTSTEP (based on 4.3BSD Unix), the username of the single installed desktop user was "me".

Re: How to become a pirate archivist

#44

Post by justsomehnguy »

totetsu wrote: post #31
chatterhead wrote: post #14

While reading this I realized that the first impression for 'Pirate Archivists' that I was exposed to were the bums in Fahrenheit 451 who memorize books so they can't be burned. I never realized that was my first true introduction to piracy. Really enjoyed the write up!

200 years ago the Grimm brothers collected tales that had been memorized and shared for generations in the German oral tradition and made them into a book of fairy tales. 70 years Walt Disney made some animated movies based on these fairy tales. Today sharing a copy of Cinderella is Piracy.

> sharing a copy of Cinderella is Piracy

with Disney artwork.

Re: How to become a pirate archivist

#45

Post by justshowpost »

imhoguy wrote: post #36

As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, ev…

But why? The current Kademlia implementation built in eMule/aMule does exactly this. Primary data advertised is file name string, there is also a support for several meta fields. Transfer proto sucks, but I don't think it really matters in the case of ultra-rare content.

Re: How to become a pirate archivist

#46

Post by chhs »

imhoguy wrote: post #36

As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, ev…

I did work on a proof of concept program to accomplish this for my own content library. It would scan a directory to find files and compare them with locally stored metadata. For v2 torrents this is trivial to do via a "pieces root" lookup, for v1 torrents it involves basically checking that each piece matches, and since pieces may not align with the file then it's not possible to guarantee that it's the same file without having all of the other files in the torrent.

I built it with libtorrent and after loading in all of the torrents (multiple TBs of data), it would promptly and routinely crashed. I couldn't find the cause of the error, it doesn't seem it was designed to run with thousands of torrents.

One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase. I haven't been able to find a publicly available database of torrent metadata. If you have an info hash then itorrents.org will give you the metadata, if it exists. I started scraping metadata via DHT announcements, but it's not exactly fast, and each client would have to do this unless they can share the database of metadata between them (I have an idea on how to accomplish this via BEP 46).

Re: How to become a pirate archivist

#47

Post by mrfinn »

I keep feeling we shouldn't accept the term "piracy" anymore. The problem, the big problem is on the so-called "legal" side, and the purpose of this system is not about retrieving authors anymore, is about some big economic groups hoarding goods (and power by doing that). But that's heavily against the common interest. I met quite a few years ago with a member of my country's senate with a solid proposition to end the "piracy" problem. Got an email asking for more info about my proposal. That was the end of it.

PS. Maybe instead "pirates", we should call ourselves "keepers".

Re: How to become a pirate archivist

#48

Post by justshowpost »

chhs wrote: post #42
londons_explore wrote: post #38

Earlier quoted context omitted.

The "v2" torrent file format allows most of this. Some clients have support already. All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere. If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.

A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm. But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there'…

This.

Also, advertising every file in torrent (of unsuspecting user) to entire world would be a HUGE privacy flaw.

Re: How to become a pirate archivist

#49

Post by 3np »

chhs wrote: post #46
imhoguy wrote: post #36

As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, ev…

I did work on a proof of concept program to accomplish this for my own content library. It would scan a directory to find files and compare them with locally stored metadata. For v2 torrents this is trivial to do via a "pieces root" lookup, for v1 torrents it involves basically checking that each piece matches, and since pieces may not align with the file then it's not possible to guarantee that it's the same file wi…

Would you mind sharing the source? Sounds like something others could build on.

Re: How to become a pirate archivist

#50

Post by justshowpost »

justsomehnguy wrote: post #44
totetsu wrote: post #31

Earlier quoted context omitted.

200 years ago the Grimm brothers collected tales that had been memorized and shared for generations in the German oral tradition and made them into a book of fairy tales. 70 years Walt Disney made some animated movies based on these fairy tales. Today sharing a copy of Cinderella is Piracy.

> sharing a copy of Cinderella is Piracy with Disney artwork .

Moreover, Grimms' copyright has been expired while Disney's copyright is set to never expire.
Post reply on HN