Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to become a pirate archivist

annas-blog.org

Post reply on HN
Print view View original

71–80 of 103 posts

Re: How to become a pirate archivist

#71

Post by throwhn0000001 »

luckylion wrote: post #65
jancsika wrote: post #15

I'm curious how Sci-hub's approach compares to the What.cd/Redacted approach. IIUC Sci-hub has scooped up science docs through a good enough UX that it was able to leverage the goodwill of science folks to upload docs (plus whatever other methods it has used to scoop up docs), and it uses a public blitzkrieg-style distribution mechanism. I.e., I guess if one had a big enough harddrive and a fast enough internet conne…

> Was any data lost between the two services? Yes, absolutely, a lot. Even though some people have automated their setup very well and have been downloading (and uploading on the newer trackers) a lot, it's just a giant amount of content, it's unlikely for any one person to have it all, and coordination was very limited back then. The birthday release of the torrent db of what.cd includes 2.6m torrents in 1.2m groups…

after wcd fell some private sftp and dc+ servers came up for about 100 of the top seeding and hoarding members who were all familiar with each other. people regularly shared new content and filled out mussing music in their collections. i dont think it exists anymore but it might. i did not have access but knew several people that did who got me content to complete music sets

the combined amount of content on those servers was probably around 100TB but most likely more

Re: How to become a pirate archivist

#72

Post by londons_explore »

chhs wrote: post #42
londons_explore wrote: post #38

Earlier quoted context omitted.

The "v2" torrent file format allows most of this. Some clients have support already. All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere. If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.

A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm. But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there'…

> there's no lookup mechanism for a "pieces root" t

I guess they'd just have to publish each pieces root in the DHT. And obviously you would let the user decide which directories to share...

I guess that'll have to wait for torrent v3...

Re: How to become a pirate archivist

#73

Post by jychang »

chhs wrote: post #42
londons_explore wrote: post #38

Earlier quoted context omitted.

The "v2" torrent file format allows most of this. Some clients have support already. All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere. If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.

A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm. But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there'…

I don't think you can solve this problem, without destroying the system by releasing too much data.

If I find a torrent of a file, and can look up all the seeders across all torrents for this file... then so can the FBI.

Re: How to become a pirate archivist

#74

Post by luckylion »

throwhn0000001 wrote: post #71
luckylion wrote: post #65

Earlier quoted context omitted.

> Was any data lost between the two services? Yes, absolutely, a lot. Even though some people have automated their setup very well and have been downloading (and uploading on the newer trackers) a lot, it's just a giant amount of content, it's unlikely for any one person to have it all, and coordination was very limited back then. The birthday release of the torrent db of what.cd includes 2.6m torrents in 1.2m groups…

after wcd fell some private sftp and dc+ servers came up for about 100 of the top seeding and hoarding members who were all familiar with each other. people regularly shared new content and filled out mussing music in their collections. i dont think it exists anymore but it might. i did not have access but knew several people that did who got me content to complete music sets the combined amount of content on those s…

Oh, certainly, with some coordination there's a lot you can achieve, and it would probably be much more than 100TB. A friend of mine had barely above 1TB seeding and iirc that was somewhere in the top 450-500, so top 100 probably had much more.

It's still a lot to store (and maintain) individually, so unless they're still going with that private exchange, each tracker end could mean someone not migrating to the next installment and their part of the library being lost. The Eye is doing a lot of public-ish work in that area (seen with very mixed feelings in the tracker scene, because apparently they're not just archivists but also jerks).

Re: How to become a pirate archivist

#75

Post by relaxing »

throwhn0000001 wrote: post #67
jancsika wrote: post #15

I'm curious how Sci-hub's approach compares to the What.cd/Redacted approach. IIUC Sci-hub has scooped up science docs through a good enough UX that it was able to leverage the goodwill of science folks to upload docs (plus whatever other methods it has used to scoop up docs), and it uses a public blitzkrieg-style distribution mechanism. I.e., I guess if one had a big enough harddrive and a fast enough internet conne…

Redacted's ratio economy is fucked. There's groups of people using high speed seedboxes that grab every new upload to build up ratio in a normal way, if you can call it that. Everyone else is left with scraps, trickling data out to whoever comes later. Or maybe you get lucky and you're the lone seeder, you get a 1:1 copy. You just better hope that was on a 24-Bit FLAC release, given how big they can get. There's many…

What saves the economy is the generous freeleech and gift token events. (Currently one running if you haven’t logged in this week.)

It all serves the purpose of getting users to be good citizens participating in the community, and not just snatch and run.

Re: How to become a pirate archivist

#76

Post by chhs »

3np wrote: post #49
chhs wrote: post #46

Earlier quoted context omitted.

I did work on a proof of concept program to accomplish this for my own content library. It would scan a directory to find files and compare them with locally stored metadata. For v2 torrents this is trivial to do via a "pieces root" lookup, for v1 torrents it involves basically checking that each piece matches, and since pieces may not align with the file then it's not possible to guarantee that it's the same file wi…

Would you mind sharing the source? Sounds like something others could build on.

The source is available here: https://github.com/chhs1/content-seeder

Re: How to become a pirate archivist

#77

Post by michaelt »

incompatible wrote: post #33
yamrzou wrote: post #2

«That secrecy, however, comes with a psychological cost. Most people love being recognized for the work that they do, and yet you cannot take any credit for this in real life.» Feels like the anonymous torrent seeder who keeps seeding a file for years just for the sake of keeping it alive. It's not easy, but some people seem to be able to derive full pleasure from accomplishing the task itself, whether recognition ha…

It's probably not that uncommon that people working for companies, governments, or criminal organizations can't talk about their work in public. One group I remember in particular are mathematicians working for the NSA, etc., who are not permitted to publish their research, then they watch as other mathematicians rediscover their work and get the credit.

NSA mathematicians still get some recognition though:

* Fellow NSA employees, your coworkers and boss

* Cash money, my personal favourite form of recognition

* Your close friends and trusted loved ones will know the broad outlines of what you're doing.

It doesn't make you world-famous but it wouldn't be as lonely as a job that needed total secrecy.

Re: How to become a pirate archivist

#78

Post by imhoguy »

londons_explore wrote: post #70
autoexec wrote: post #66

Earlier quoted context omitted.

I have seen people obviously sharing their entire hard drive on P2P platforms, and it's usually been someone's C drive and a massive security issue. I don't think some clients do enough to make it clear to people that they need to be careful not to share more than they intend to or should.

On modern windows, ideally there are no sensitive files accessible to a user program, because everything should be in the secret store/protectedstorage. Obviously a lot of programs still store cached session keys and passwords in plaintext somewhere in your user directory though...

and assuming nobody puts their passwords.txt on desktop and keeps unprotected wallet.dat around.

Re: How to become a pirate archivist

#79

Post by Fnoord »

jtsuken wrote: post #21
userbinator wrote: post #8

Earlier quoted context omitted.

And from your addendum: So, use random usernames on the computers you use for this stuff, in case you misconfigure something. ...or a username that is so common as to be meaningless, like "user", "Administrator", or even "root".

The next step is to change your legal name to Zhang Wei or at least a John Smith.

差不多 (chabuduo / chàbuduō)

Re: How to become a pirate archivist

#80

Post by eimrine »

jancsika wrote: post #15

I'm curious how Sci-hub's approach compares to the What.cd/Redacted approach. IIUC Sci-hub has scooped up science docs through a good enough UX that it was able to leverage the goodwill of science folks to upload docs (plus whatever other methods it has used to scoop up docs), and it uses a public blitzkrieg-style distribution mechanism. I.e., I guess if one had a big enough harddrive and a fast enough internet conne…

> What.cd/Redacted approach seems to use Bittorrent ratios to create a private-tracker economy. New users get a few gigs free download on joining.

Isn't that "bittorrent ratio" easy to cheat? I remember a good old times where I had to download some popular files just for ratio, then (beginning of 10's everybody starts to cheat and some of the biggest trackers turned to forever free leach.

Post reply on HN