Archivists Are Trying to Make Sure LibGen Never Goes Down
1–10 of 270 posts
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#2This is an extremely important effort. The LibGen archive contains around 32 TBs of books (by far the most common being scientific books and textbooks, with a healthy dose of non-STEM). The SciMag archive, backing up Sci-Hub, clocks in at around 67 TBs [0]. This is invaluable data that should not be lost. If you want to contribute, here's a few ways to do so.
If you wish to donate bandwidth or storage, I personally know of at least a few mirroring efforts. Please get in touch with me over at legatusR(at)protonmail(dot)com and I can help direct you towards those behind this effort.
If you don't have storage or bandwidth available, you can still help. Bookwarrior has requested help [1] in developing an HTTP-based decentralizing mechanism for LibGen's various forks. Those with experience in software may help make sure those invaluable archives are never lost.
Another way of contributing is by donating bitcoin, as both LibGen [2] and The-Eye [3] accept donations.
Lastly, you can always contribute books. If you buy a textbook or book, consider uploading it (and scanning it, should it be a physical book) in case it isn't already present in the database.
In any case, this effort has a noble goal, and I believe people of this community can contribute.
P.S. The "Pirate Bay of Science" is actually LibGen, and I favor a title change (I posted it this way as to comply with HN guidelines).
[0] http://185.39.10.101/stat.php
[1] https://imgur.com/a/gmLB5pm
[2] bitcoin:12hQANsSHXxyPPgkhoBMSyHpXmzgVbdDGd?label=libgen, as found at http://185.39.10.101/, listed in https://it.wikipedia.org/wiki/Library_Genesis
[3] Bitcoin address 3Mem5B2o3Qd2zAWEthJxUH28f7itbRttxM, as found in https://the-eye.eu/donate/. You can also buy merchandising from them at https://56k.pizza/.
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#3There is a huge amount of duplication there (i.e. books that have many scans), I wonder if it would be better to tackle that versus doing a straight backup.
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#4There is a huge amount of duplication there (i.e. books that have many scans), I wonder if it would be better to tackle that versus doing a straight backup.
This is a downside of Libgen: duplicate uploads, missing or erroneous metadata. You start wishing that there was at least some curation of the collection, so it could approach the quality of an academic library catalogue as many users are usedto. But I guess the people behind Libgen want to keep the number of people with database edit rights small. (When you upload a book, you yourself can edit the metadata for that book for 24 hours, but you cannot go through the rest of LibGen's database and make corrections.)
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#5one of the next interplanetary or Interstellar Probe should carry a copy of the sci-hub torrent in some kind of permanent storage
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#6There is a huge amount of duplication there (i.e. books that have many scans), I wonder if it would be better to tackle that versus doing a straight backup.
There are groups behind data curation as well, though it is much harder. LibGen sees an addition rate of about 230 GBs per month, while SciMag's is around 1.10 TBs per month. We should expect those numbers to increase in the future. The man-hours required to curate those database may very well cost much more than the storage and bandwidth required to store duplicates and incorrectly tagged files. In any case, as I said, there are people seriously interested in curating the LibGen database, though most efforts I know of are still in the earliest stages.
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#7Is there a way to just download the whole 32TB to your own machine? I see a ton of mirrors but the content seems to be highly fragmented between them
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#8There is a huge amount of duplication there (i.e. books that have many scans), I wonder if it would be better to tackle that versus doing a straight backup.
This is a downside of Libgen: duplicate uploads, missing or erroneous metadata. You start wishing that there was at least some curation of the collection, so it could approach the quality of an academic library catalogue as many users are usedto. But I guess the people behind Libgen want to keep the number of people with database edit rights small. (When you upload a book, you yourself can edit the metadata for that…
Maybe they should consider a system where users can suggest tags/metadata or flag erroneous data that can be reviewed and allowed by a select few?
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#9one of the next interplanetary or Interstellar Probe should carry a copy of the sci-hub torrent in some kind of permanent storage
Do we have anything rated for a few millennia of interstellar radiation besides etched gold plates?
Re: Archivists Are Trying to Make Sure LibGen Never Goes Down
#10One could use FAANG data centers to host them for free, it would be really great