Live data from Hacker News

Archivists Are Trying to Make Sure LibGen Never Goes Down

vice.com

31–40 of 270 posts

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#31

One could use FAANG data centers to host them for free, it would be really great

Look at the google books project. That got shutdown real hard due to copyright issues and litigation after they invested a ton of money in digitizing some of the most valuable library collections in the world.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#32

Earlier quoted context omitted.

I think the duplication issue is probably overstated. I doubt tackling that would shave off more than 20% of the total backup size.

Speaking from personal experience, I usually see several results for any search. Granted, there's a big selection bias there, but 20% seems way too small.

To be fair for textbooks at least I often see several results but often of different editions (1x edition 1, 2x edition 2, 1x edition 3 etc.). In some cases I think it's worthwhile keeping the different additions around, unless it becomes a huge burden.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#33
post #5

one of the next interplanetary or Interstellar Probe should carry a copy of the sci-hub torrent in some kind of permanent storage

Storing that amount of information in a way that an unknown alien species would be able to read (even assuming technical expertise greater than our own) is a huge problem. Keep in mind that they don't know our written or computational language and there's nothing about our technology that is inherently self-explaining/obvious. Even the assumption that they'd use binary computers (rather than trinary, or other technol…

> an unknown alien species

Not necessarily for aliens ... but why not keep a backup in a safe place outside the dangers of earthlings ?

OTOH, I think a sufficiently advanced alien intelligence will be able to decipher the information structures we use regardless of differences in technology. It's possible though there will be missing links in that archive, which will need to be supplemented with a primary secondary and high school curriculim.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#34

Earlier quoted context omitted.

I think the duplication issue is probably overstated. I doubt tackling that would shave off more than 20% of the total backup size.

Speaking from personal experience, I usually see several results for any search. Granted, there's a big selection bias there, but 20% seems way too small.

Because you or anyone is most likely to search for relatively popular books. So those books will have a multiple copies. But for every popular book, there are many unpopular, but still useful books, that only have a single copy.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#35
post #25

Related: Looking at harddisk cost per terabyte, quite often extern drives are cheaper than internal ones. For example right now in Germany I can get a WD 8TB USB 3.0 drive for 135€ but the cheapest internal 8TB drive costs 169€. Any idea why? It's puzzling.

I noticed this yesterday while shopping for cyber Monday deals. If you want to load up a server with drives, perhaps the external drives can be removed from their cases and used internally?

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#36
post #25

Related: Looking at harddisk cost per terabyte, quite often extern drives are cheaper than internal ones. For example right now in Germany I can get a WD 8TB USB 3.0 drive for 135€ but the cheapest internal 8TB drive costs 169€. Any idea why? It's puzzling.

It has been like this for years. You'll often see people refer to "shucking" them, taking the drives out to use in a NAS.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#37
post #25

Related: Looking at harddisk cost per terabyte, quite often extern drives are cheaper than internal ones. For example right now in Germany I can get a WD 8TB USB 3.0 drive for 135€ but the cheapest internal 8TB drive costs 169€. Any idea why? It's puzzling.

My best guess would be that more people buy external drives than internal, and those that "manufacture" external drives (ie - buy internal drives and repackage them) purchase in larger volumes than those that sell bare internal drives.

Of course, that wouldn't explain the difference between a WD external drive and that same drive as an internal drive - assuming that WD actually manufactures both (and doesn't just license the name provided the 3rd party uses their drives)...

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#38
post #3

There is a huge amount of duplication there (i.e. books that have many scans), I wonder if it would be better to tackle that versus doing a straight backup.

I think the duplication issue is probably overstated. I doubt tackling that would shave off more than 20% of the total backup size.

20% is not insignificant.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#39
post #23
post #11

Earlier quoted context omitted.

There are ways to do so. The archive is made up of many, many torrents (I believe it's a monthly if not biweekly update of the database). If you have the storage/bandwidth availability for the whole 32TBs, please get in touch and I may be able to help you get the whole deal without too much hassle. Otherwise, just pick some torrents (it would be best to pick them based on torrent health, but they are so many to check…

I'm pretty surprised by the lack of seeders. Out of the 2438 torrents listed, a third have 0 seeders, another third have 1 seeder, and all but 5 have less than 10. Hopefully the publicity boosts those numbers.

From what I've heard a good chunk of people rotate their seeds for LibGen because their seedboxes can't handle all the connections for every torrent at once.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#40
post #31

One could use FAANG data centers to host them for free, it would be really great

Look at the google books project. That got shutdown real hard due to copyright issues and litigation after they invested a ton of money in digitizing some of the most valuable library collections in the world.

It’s incredibly sad: https://www.theatlantic.com/technology/archive/2017/04/the-t...
Post reply on HN