Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Archivists Are Trying to Make Sure LibGen Never Goes Down

vice.com

Post reply on HN
Print view View original

91–100 of 270 posts

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#91

Post by burtonator »

I've thought that we could potentially build an end to end encrypted datastore within Polar and possibly add IPFS support to potentially help with this issue.

Here's a blog post about our datastores for some background.

https://getpolarized.io/2019/03/22/portable-datastores-and-p...

... essentially Polar is a PDF manager and knowledge repository for academics, scientists, intellectuals, etc.

One secondary challenge we have is allowing for sharing of research but I'd like to do it in a secure and distributed manner.

Some of our users are concerned about their eBooks being stored unencrypted and while for the majority of our users this will never be a problem I can see this being an issue in countries with political regimes that are hostile to open research.

In the US we have an issue of researchers being harassed over climate change btw. Having a way to encrypt your knowledge repository (ebooks) would help academic freedom as your employer or government couldn't force you to give them your repository.

But what if we went beyond this and provided a way to ADD documents to the repository from a site like LibGen?

Then we'd have the ability to easily, with one click, encrypt the document (end to end) and added it to our repository.

If we can add support for Polar to allow colleagues to share directly, this would be a virtual mirror of LibGen.

Alice could add books b1, b2, b3 to their repo, they could then share with Bob, only he would be able to see b1, b2, b3, then they would generate a shared symmetric key to share the books.

No 3rd party (including me) would have any knowledge what's going on.

I'm going to assume our users are not going to do anything nefarious or pirate any books. I'm also certain that they're confirming to the necessary laws ...

The challenge though is that while we'd be able to have a mirror of LibGen and more material, it would be a probabilistic mirror - I'm sure we'd have like 60% of it but the obscure material wouldn't be mirrored.

Right now our datastores support just local disk, and Firebase (which is Google Cloud basically). While we would encrypt the data end to end in Google Cloud I can totally understand why users might not like to use that platform.

One major issue is China where it's blocked.

Something like IPFS could go a long way to solving this but it's still very new and I haven't hacked on it much.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#92

Post by eitland »

stavros wrote: post #70
traverseda wrote: post #54

Earlier quoted context omitted.

In my experience ipfs doesn't actually work. I'd love to be proven wrong, but the reason why nobody uses ipfs even when it seems like a great fit is bect it's not really usable.

This is my experience as well. In theory, IPFS is exactly the right thing for LibGen, but in practice I consider it unusable.

FWIW: StavrosK has actually been putting some serious effort into making IPFS accessible.

See here for example: https://news.ycombinator.com/item?id=16521385

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#93

Post by saalweachter »

jandrese wrote: post #84
saalweachter wrote: post #82

Earlier quoted context omitted.

The volume isn't a real problem, but nearly half a cubic meter of gold should also have a mass of ~7500 kilograms. You're also looking at a cost of ~$350 million, US. (Which isn't necessarily impossible , but still a lot , especially for an interstellar probe.)

7.5 metric tons is well within the weight limit for a Falcon 9, so add about $57m for the launch costs for a total project cost of probably around $500m. That puts it in the realm of the Voyager probes for total cost.

7.5 metric tons is also over ten times the mass of each Voyager probe.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#94

Post by StavrosK »

anchpop wrote: post #89
stavros wrote: post #70

Earlier quoted context omitted.

This is my experience as well. In theory, IPFS is exactly the right thing for LibGen, but in practice I consider it unusable.

How would that work with adding new books and metadata? IPFS archives are immutable, right? I think something like Dat might be better because the people with the secret keys could update the archive and everyone else would automatically seed the updated version

You can just have it pin an IPNS CID, or you can publish a new hash for people to pin. There are ways.

That said, maybe Dat would be better, especially if it works well.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#95

Post by [deleted] »

eitland wrote: post #92
stavros wrote: post #70

Earlier quoted context omitted.

This is my experience as well. In theory, IPFS is exactly the right thing for LibGen, but in practice I consider it unusable.

FWIW: StavrosK has actually been putting some serious effort into making IPFS accessible. See here for example: https://news.ycombinator.com/item?id=16521385

[deleted]

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#96

Post by StavrosK »

eitland wrote: post #92
stavros wrote: post #70

Earlier quoted context omitted.

This is my experience as well. In theory, IPFS is exactly the right thing for LibGen, but in practice I consider it unusable.

FWIW: StavrosK has actually been putting some serious effort into making IPFS accessible. See here for example: https://news.ycombinator.com/item?id=16521385

Thank you, I really hope IPFS improves.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#97

Post by zozbot234 »

fghtr wrote: post #75

Are there any i2p torrents? I guess anonymity might be helpful if I want to mirror/seed this data...

I assume anyone could simply seed the "official" torrents via i2p? Not sure how that system actually works, it's interesting for sure but a lot less well-known than the alternatives.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#98

Post by jandrese »

saalweachter wrote: post #93
jandrese wrote: post #84

Earlier quoted context omitted.

7.5 metric tons is well within the weight limit for a Falcon 9, so add about $57m for the launch costs for a total project cost of probably around $500m. That puts it in the realm of the Voyager probes for total cost.

7.5 metric tons is also over ten times the mass of each Voyager probe.

Thank SpaceX for slashing launch costs and me for not factoring in much of a second stage to boost that mass out of orbit.

But I was also extremely generous with the thickness in the first post. In real life they would almost certainly be closer to .05mm than 1.2mm, and probably not made out of solid gold.

The point was to show that even with some rather pessimistic assumptions the project was within human scale and even had some precedent.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#99

Post by weinzierl »

sanxiyn wrote: post #57

Yongle Encyclopedia was a similar project of the 15th century China. It was the largest encyclopedia in the world for 600 years until surpassed by Wikipedia. Alas, Yongle Encyclopedia is almost completely lost now. Archiving is harder than you think. https://en.wikipedia.org/wiki/Yongle_Encyclopedia

I read the Wikipedia article about it and the sad thing is that the majority of the Yongle Encyclopedia seem to have been destroyed only in quite recent times.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#100

Post by jerf »

Someone1234 wrote: post #22
buboard wrote: post #5

one of the next interplanetary or Interstellar Probe should carry a copy of the sci-hub torrent in some kind of permanent storage

Storing that amount of information in a way that an unknown alien species would be able to read (even assuming technical expertise greater than our own) is a huge problem. Keep in mind that they don't know our written or computational language and there's nothing about our technology that is inherently self-explaining/obvious. Even the assumption that they'd use binary computers (rather than trinary, or other technol…

An idea I've seen is including messages at several levels. At the outermost level you describe in very basic format how to build a magnifying glass. From there you have diagrams that are legible that describes how to build a microscope. From there you have more than enough space to describe the basics of what else is in there and to start describing your language. I'm thinking optical storage in a clear rock of some sort, as has already been prototyped.

If you assume motivated readers and human-level intelligence, you could end up with good results. It might take a decade or three, and a lot of mental firepower, but they could get there.

(The outer layer is the hardest, since our information density is lowest. Our "description of how to build a magnifying glass" might cover just the basic optics of curved glass and a very basic description of how to get to glass and how to curve it correctly, leaving a lot of the details up to the finder. After all, we did it without help. We're not so much trying to solve this problem for the finder as help them on their way.)

So, before jumping in to argue, remember I'm stipulating decades of dedicated effort by presumably an interested consortium of... whatever they are. I think we can safely stipulate an amount of effort at least as large as our society has dedicated to, say, Linear A and B, or the Voynich manuscript. I'm not trying to spec "Ugh wanders out of the jungle, sees our pretty rock, and personally has a 20th century civilization up and running in 10 years" or anything crazy.

Post reply on HN