Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Archivists Are Trying to Make Sure LibGen Never Goes Down

vice.com

Post reply on HN
Print view View original

191–200 of 270 posts

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#191

Post by EthanHeilman »

mrob wrote: post #187
asdff wrote: post #170

Earlier quoted context omitted.

Would an SSD even function after 1000 years? Unless sealed, I imagine ambient moisture would do a number inside the drive. The same is true for books of course, but we still have 1000 year old books that have lasted by sitting on a shelf in churches and temples, etc., without any specific care until recent history. The nice part of a book in an apocalyptic scenario is that you can copy it even if you don't know the l…

It doesn't really matter if the SSD as a whole still works, because after 1000 years you'll never recover the data via the normal interface. Modern MLC flash is often specified for less than 1 year data retention, and even SLC is unlikely to make it to 1000 years. Attempting to read it will only make things worse ("read disturb"). The best hope of saving the data is with some future nanotech that directly probes each…

I would assume they would read the SSD not by powering it on and plugging it into to a computer but by disassembling it and physically imaging the physical structure. This would also bypass the all the write leveling infrastructure allowing them to recover deleted data. It reminds me of the current techniques of using x-rays to read writing on the odd scraps of paper used to bind a book [0].

[0]: "X-rays reveal 1,300-year-old writings inside later bookbindings" https://www.theguardian.com/books/2016/jun/04/x-rays-reveal-...

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#192

Post by abawany »

toomuchtodo wrote: post #27
canuckintime wrote: post #24

Earlier quoted context omitted.

> Lastly, you can always contribute books. If you buy a textbook or book, consider uploading it (and scanning it, should it be a physical book) in case it isn't already present in the database. There's no easy solution for scanning physical books, is there?

There are providers [1] that will destructively scan the book for you and return a PDF. If you want to preserve the book, you're stuck using a scanning rig [2]. The Internet Archive will also non-destructively scan as part of Open Library [3], but they only permit one checkout at a time of scanned works, and the latency can be high between sending them a book and it becoming available. FYI, 600 DPI is preferred for a…

I will add a vote for bookscan.us, which I have been using since 2013 or so. Very reasonable prices and great service.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#193

Post by alteria »

gdy wrote: post #163
mikorym wrote: post #145

Earlier quoted context omitted.

I am not an expert on the topic, but I believe that in the former Soviet Union it was common between mathematicians to pass around preprints (a la arXiv). These then perculated through to the West. I think it had to do with the USSR and their restrictive (if we are being euphemistic) policies towards academics.

"the USSR and their restrictive (if we are being euphemistic) policies towards academics." What do you mean?

Their policies were for more than "restrictive" is how I'm reading it

See [1]

[1] https://en.wikipedia.org/wiki/Suppressed_research_in_the_Sov...

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#194

Post by asdefghyk »

qmmmur wrote: post #144
EthanHeilman wrote: post #107

Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material becam…

Pretty much everyone in a tech job could afford to buy 40TB of storage at home, or remotely and mirror the entire repo. I think that given this low barrier of entry if you can afford to help preserve the information then you can and probably should. Even if a small amount do it it's more points of recovery.

Id say.... in the day .... anyone could afford to buy a single floppy disk and store files on it. But how many actually did and how many are actually recoverable. Lots probably got thrown out in intervening years.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#195

Post by 8bitsrule »

sanxiyn wrote: post #57

Yongle Encyclopedia was a similar project of the 15th century China. It was the largest encyclopedia in the world for 600 years until surpassed by Wikipedia. Alas, Yongle Encyclopedia is almost completely lost now. Archiving is harder than you think. https://en.wikipedia.org/wiki/Yongle_Encyclopedia

WP says that it was never printed for the general public. Hmmm. Had it been (parts duplicated, say, at hundreds of sites), most of it would probably have survived.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#196

Post by [deleted] »

walrus01 wrote: post #109
turc1656 wrote: post #104

I don't see anyone having mentioned the possibility of posting this data to Usenet at all - at minimum for archival purposes which should be good for ~8-9 years. That way at least the data isn't lost. With so many of those torrents have 0 or 1 seed, this is a serious risk I think, despite the comments elsewhere about people rotating what they seed. I realize that doesn't solve the access problem for most people as mo…

Two thoughts on that. Encoding it to a text format with CRC data for posting to usenet is highly inefficient in terms of data storage. And 33TB of stuff is not going to be retained for 8-9 years, the last I checked due to the huge volume of binaries traffic, the major commercial usenet feed providers have at most 6-9 months of retention for the major binary groups. Beyond that it becomes cost prohibitive for them in…

[deleted]

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#197

Post by EthanHeilman »

TheOtherHobbes wrote: post #179
EthanHeilman wrote: post #107

Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material becam…

This is not a solvable problem without technological continuity, or some unimaginably smart technology we can't imagine today. If you found a mysterious archive object and had no idea what it was - CD-R, hard drive, SSD, whatever - not only would you have to reinvent an entire hardware reader around it, you would also have to work out the file structure, extract the data (some of which could be damaged), and reverse…

Take a CD-R of some MP3 with English language file names stored on a FAT32 filesystem for example. Assume the reflective layer didn't rust since it was abandoned in a dry climate and our future archaeologist has access to roughly modern levels of technology.

1. Even if the CD-R has been crushed and shattered you could use a modern and cheap microscope to read continuous pits and lands off the disk [0,1]. It would be clear to anyone familiar with information theory how to translate the pits and lands to a series of set of arbitrary symbols which encode data.

2. This data would at first be meaningless. However the mathematical relationships of a simple error correcting code would stand out. This would allow them recover corrupted data. Once the error correcting code was stripped out they have a transcript of the raw data.

3. They would notice a pattern in the data. There would be long high entropy regions and then very short low entropy regions. They would probably notice that some of the low entropy regions had every 8-th bit set to zero (ASCII) and if taken in 8-bit chunks these regions had the roughly the same number of symbols as in the latin alphabet. If they were familiar with English they might quickly decode these regions using letter frequency correspondence with another English text.

4. The high entropy regions would be far harder to decode. However these future archaeologists would be faced with the obvious data patterns of frames of an MP3. Decoding the first MP3 would be a serious project involving many institutions over many years but once it was done it would allow the decoding of all artifacts that use the MP3 and related encoding formats. Possibly someone would find a "rosetta file" [2], a disk that contained both a .wav file and an encoded MP3 of the same song. More likely someone would find an MP3 player and then reverse engineer the decoding algorithm.

[0]: "Being able to see the tracks and bits in a CD-ROM" https://superuser.com/questions/870776/being-able-to-see-the...

[1]: "CD-ROM Under the Microscope" https://www.youtube.com/watch?v=RZUxemOE07Q

[2]: https://en.wikipedia.org/wiki/Rosetta_Stone

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#198

Post by jdnenej »

Symbiote wrote: post #72
guidoism wrote: post #69

Earlier quoted context omitted.

For important archives like this maybe we need some sort of turn-key solution for the masses? Like a Raspberry Pi image that maintains a partial mirror. Imagine if one could by a RPi and external HD, burn the image, and connect it to some random wifi network (at home, at work, at the library, etc).

I'm not hosting a copy of this at work (where we easily have 32TB on old hardware) since distributing it is copyright infringement. The same goes for my home connection.

Most people don't care. The chance anything at all bad will happen is so incredibly low.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#199

Post by jdnenej »

rollinDyno wrote: post #157
miki123211 wrote: post #43

The new architecture of pirate sites, what I call the Hydra architecture, seems pretty interesting to me. There isn't a single site hosting the content, but a group of mirrors freely exchanging data between one another. In case some of them go down, the other ones still remain and new ones can appear, copying data from the remaining mirrors. This is like a hydra that grows two heads every time you chop one off. It's…

I worry that if this system becames permanent, one in which it is practically impossible to stop piracy, followed by the loss of traditional incentives we might find ourselves in a place where no motivated investor will break even when producing quality and innocuous content.

Most of the stuff on scihub was funded by tax dollars

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#200

Post by erikpukinskis »

frobozz wrote: post #165
EthanHeilman wrote: post #107

Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material becam…

> I do wonder wither digital or analogue formats are better able to survive into the distant future. There are 5000 year old clay tablets we can still read. There are centuries old documents on paper, vellum etc. that we can still read. I personally have decades-old paper documents I can easily read, and a box of floppies I can't. It's not just a problem of unreadable physical media, I have a database file on a perfe…

No one is proposing we use floppy disks.

Redundant, shared servers ARE a forever solution. Making sure your data is one one of the ones that makes it seems like a vastly easier proposition to me than writing data to clay tablets and trying to keep those from ending up in a dump somewhere.

Post reply on HN