Live data from Hacker News

Archivists Are Trying to Make Sure LibGen Never Goes Down

vice.com

141–150 of 270 posts

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#141

Earlier quoted context omitted.

Scanning with your phone is getting easier. At a minimum you can take a pic of each of the pages. Software can clean up the images, sorta. It's not ideal but it's better than nothing.

I found vFlat to be magical in cleaning up book scan images you took with your phone. https://play.google.com/store/apps/details?id=com.voyagerx.s...

>This app is incompatible with your device.

my disappointment is immeasurable and my day is ruined

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#142

Earlier quoted context omitted.

> at no greater or lesser annual failure rate than the expensive enterprise hard drives. I've read these reports as well, but I can say that it's not my experience (we've gone through a few rounds of shucking at the Internet Archive, for economy and in one case necessity after the 2011 Thailand floods pinched the supply chain). Our raw failure rates on shucked drives are significantly higher, and the drives themselve…

From reddit.com/r/datahoarder I don't believe I've seen a single instance of 8, 10 or 12TB consumer USB3 drives coming out of the plastic case as a model that is shingled recording. The average consumer trying to copy many dozens of GB onto an external drive would not tolerate SMR write performance.

Some of the disks in the market over the last few years were not well-labeled in terms of revealing their SMR internals, and use larger media caches to disguise write performance issues (at least, until they don't). For example, the Seagate STGY8000400 is an external 8TB SMR drive. But the industry as a whole is moving to host-managed SMR, so hopefully that specific issue with external disks will soon go away.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#143

Earlier quoted context omitted.

You are correct, the end user experience has improved tremendously, I tried the desktop bundle the other day and it was indeed very easy to get started with. > pinning services like Infura, Temporal, and Pinata have helped abstract some of those challenges I wonder if you omitted Eternum on purpose :P (For context, I created and run Eternum, and that experience is mostly where my opinion of IPFS comes from.)

Gotcha! Thank you! Running a pinning service definitely still has rough edges =/ but I know the Infura team recently open sourced some of the tooling they built to make it a bit easier: https://blog.infura.io/new-tools-for-running-ipfs-nodes-196d... . Might help others who are self hosting a large chunk of data on a persistent node too... If you ever want to chat about how we can make pinning services on IPFS easier…

Ah, thanks for that link, that would have come in handy a few weeks ago when I migrated the node to a new server.

I would love to chat. My #1 request is to make pinning asynchronous, and generally improve pinning performance. I think that's most of my frustration, followed by slow DHT resolves, followed by large resource usage by the node.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#144

Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material becam…

Pretty much everyone in a tech job could afford to buy 40TB of storage at home, or remotely and mirror the entire repo. I think that given this low barrier of entry if you can afford to help preserve the information then you can and probably should. Even if a small amount do it it's more points of recovery.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#145
post #130

Libgen is one of the greatest contributors to scientific productivity worldwide, possibly beaten only by Sci-Hub. Just about everybody in academia knows about it. If it ever vanished, some of us could probably still get by trading files from person to person, but nothing could be as perfect as what we got now.

> possibly beaten only by Sci-Hub Today I learned that Library Genesis is actually "powered by Sci-Hub" as its primary source. So I guess they're sister projects by similarly minded people (who seem to be mostly/originally based in Slavic countries, which I find interesting culturally - perhaps it's due to a looser legal environment + activist academics?). > Just about everybody in academia knows about it. That reall…

I am not an expert on the topic, but I believe that in the former Soviet Union it was common between mathematicians to pass around preprints (a la arXiv). These then perculated through to the West. I think it had to do with the USSR and their restrictive (if we are being euphemistic) policies towards academics.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#146
post #70

Earlier quoted context omitted.

This is my experience as well. In theory, IPFS is exactly the right thing for LibGen, but in practice I consider it unusable.

Sorry to hear about your bad experience, StavrosK. I think this perspective really depends how you're trying to use IPFS. For example, the ease of use of running a local IPFS node has improved a ton with IPFS-desktop & companion, and tools like ipfs-cohost ( https://github.com/ipfs-shipyard/ipfs-cohost ) also improve usability and tooling for shared community hosting. I think this has actually seen a ton of progress…

Pinning services are nice, but the idea of pinning services is a bit antithetical to the basic philosophy of p2p. If the only way to make something available is e.g. putting something on pinata, I might as well put it on S3.

The basic problem is that the DHT is currently not working, and IPFS is using the DHT in a very demanding way compared to, say, bittorrent or DAT.

I know that there are some fixes in the works, but the next releases really need to solve the DHT problem, otherwise no amount of usability improvements is going to matter...

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#148

Earlier quoted context omitted.

7.5 metric tons is also over ten times the mass of each Voyager probe.

Thank SpaceX for slashing launch costs and me for not factoring in much of a second stage to boost that mass out of orbit. But I was also extremely generous with the thickness in the first post. In real life they would almost certainly be closer to .05mm than 1.2mm, and probably not made out of solid gold. The point was to show that even with some rather pessimistic assumptions the project was within human scale and…

Yeah, it's all within the realm of possibility, with too many unknowns (eg, what is a realistic information density) to really easily say how easy or hard it is.

Re: Falcon 9's, Voyager was lifted on a Titan IIIE, which had a LEO payload of 15,300 kg, compared to 22,800 kg for the Falcon 9 to LEO. Assuming Voyager was near the capacity of what could be ejected from the Solar System by a Titan IIIE, you'd need to send up ~7 Falcon 9's to Voltron together in orbit.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#150
post #144

Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material becam…

Pretty much everyone in a tech job could afford to buy 40TB of storage at home, or remotely and mirror the entire repo. I think that given this low barrier of entry if you can afford to help preserve the information then you can and probably should. Even if a small amount do it it's more points of recovery.

Storage isn't hard, but downloading 40 Tb can be a problem. Are there any arrangements for physical distribution (of the "truck loaded with USB drives" variety)?
Post reply on HN