Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Archivists Are Trying to Make Sure LibGen Never Goes Down

vice.com

Post reply on HN
Print view View original

131–140 of 270 posts

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#131

Post by traverseda »

StavrosK wrote: post #106
traverseda wrote: post #102

Earlier quoted context omitted.

It's not really an accessibility issue so much as a performance one. If it was a reasonable alternative to something like the rsync daemon I'd use it all the time. Unfortunately the performance issues and overhead is just too much.

IPFS isn't really an alternative to rsync, but the rest of your point stands.

Not in the general case, no. But one of the major uses for rsync in synchronizing mirrors, and it could very much be a replacement for rsync in that particular niche.

For example mirroring project gutenburg.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#132

Post by momack2 »

stavros wrote: post #70
traverseda wrote: post #54

Earlier quoted context omitted.

In my experience ipfs doesn't actually work. I'd love to be proven wrong, but the reason why nobody uses ipfs even when it seems like a great fit is bect it's not really usable.

This is my experience as well. In theory, IPFS is exactly the right thing for LibGen, but in practice I consider it unusable.

Sorry to hear about your bad experience, StavrosK.

I think this perspective really depends how you're trying to use IPFS. For example, the ease of use of running a local IPFS node has improved a ton with IPFS-desktop & companion, and tools like ipfs-cohost (https://github.com/ipfs-shipyard/ipfs-cohost) also improve usability and tooling for shared community hosting. I think this has actually seen a ton of progress and end consumer usability has improved significantly in the past year (and is now even coming out-of-the-box in browsers like Brave and Opera!)

I definitely hear that running a beefy IPFS node for local hosting/pinning still needs work, but pinning services like Infura, Temporal, and Pinata have helped abstract some of those challenges from individual applications like this. From a developer perspective, there are a lot of performance improvements for adding, data transfer, and content resolution coming down the line very soon (https://github.com/ipfs/go-ipfs/issues/6776), and there's also been a lot of work improving the ipfs gateways and docs to support the dev community better. I definitely think there is still lots of room for improvement - but also lots of progress to recognize in making IPFS usable to exactly these sorts of applications. Check out qri,io - they're doing collaborative dataset hosting like this and it's pretty slick!

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#133

Post by lioeters »

lovecg wrote: post #40
woofcat wrote: post #31

Earlier quoted context omitted.

Look at the google books project. That got shutdown real hard due to copyright issues and litigation after they invested a ton of money in digitizing some of the most valuable library collections in the world.

It’s incredibly sad: https://www.theatlantic.com/technology/archive/2017/04/the-t...

> Somewhere at Google there is a database containing 25 million books and nobody is allowed to read them.

Indeed, what an intellectual tragedy..

> In August 2010, Google put out a blog post announcing that there were 129,864,880 books in the world. The company said they were going to scan them all.

That seems like a surprisingly "small" number.

Well, in trying to picture a physical library with 130 million books, maybe that's a realistic estimate. But compared to, say, the recently discovered data hoard of more than 2 billion online identities, it's miniscule.

SciHub and LibGen are truly the modern-day Library of Alexandria. The fact that they're being called "Pirate Bays of Science" - and that providing free and open access to all books in the world is illegal - just goes to show that our civilization's priorities are misdirected.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#134

Post by StavrosK »

traverseda wrote: post #131
StavrosK wrote: post #106

Earlier quoted context omitted.

IPFS isn't really an alternative to rsync, but the rest of your point stands.

Not in the general case, no. But one of the major uses for rsync in synchronizing mirrors, and it could very much be a replacement for rsync in that particular niche. For example mirroring project gutenburg.

Ah, yes, you mean pinning a specific set, you're right. Unfortunately I've found that the way the daemon does pinning currently doesn't lend itself to that use case (a single unavailable file will stall the pin for hours).

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#135

Post by StavrosK »

momack2 wrote: post #132
stavros wrote: post #70

Earlier quoted context omitted.

This is my experience as well. In theory, IPFS is exactly the right thing for LibGen, but in practice I consider it unusable.

Sorry to hear about your bad experience, StavrosK. I think this perspective really depends how you're trying to use IPFS. For example, the ease of use of running a local IPFS node has improved a ton with IPFS-desktop & companion, and tools like ipfs-cohost ( https://github.com/ipfs-shipyard/ipfs-cohost ) also improve usability and tooling for shared community hosting. I think this has actually seen a ton of progress…

You are correct, the end user experience has improved tremendously, I tried the desktop bundle the other day and it was indeed very easy to get started with.

> pinning services like Infura, Temporal, and Pinata have helped abstract some of those challenges

I wonder if you omitted Eternum on purpose :P

(For context, I created and run Eternum, and that experience is mostly where my opinion of IPFS comes from.)

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#136

Post by Hitton »

saalweachter wrote: post #9
buboard wrote: post #5

one of the next interplanetary or Interstellar Probe should carry a copy of the sci-hub torrent in some kind of permanent storage

Do we have anything rated for a few millennia of interstellar radiation besides etched gold plates?

I think millipede memory[0] could have managed that, too bad the technology got shelved.

[0]: https://en.wikipedia.org/wiki/Millipede_memory

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#137

Post by momack2 »

traverseda wrote: post #131
StavrosK wrote: post #106

Earlier quoted context omitted.

IPFS isn't really an alternative to rsync, but the rest of your point stands.

Not in the general case, no. But one of the major uses for rsync in synchronizing mirrors, and it could very much be a replacement for rsync in that particular niche. For example mirroring project gutenburg.

Better rsync support is actually in the works (though probably still a ~quarter out to land end to end in go-ipfs). The two main components are improving ipfs "mount" support, and adding metadata support into unixfs so we can only update modified files. See details here: https://github.com/ipfs/team-mgmt/blob/master/OKR/PACKAGE-MA... -- mount currently has IPNS read support but write support needs an owner to get it over the finish line and unixfsv1.5 (with metadata) should be landing in js-ipfs later this week!

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#138

Post by momack2 »

StavrosK wrote: post #135
momack2 wrote: post #132

Earlier quoted context omitted.

Sorry to hear about your bad experience, StavrosK. I think this perspective really depends how you're trying to use IPFS. For example, the ease of use of running a local IPFS node has improved a ton with IPFS-desktop & companion, and tools like ipfs-cohost ( https://github.com/ipfs-shipyard/ipfs-cohost ) also improve usability and tooling for shared community hosting. I think this has actually seen a ton of progress…

You are correct, the end user experience has improved tremendously, I tried the desktop bundle the other day and it was indeed very easy to get started with. > pinning services like Infura, Temporal, and Pinata have helped abstract some of those challenges I wonder if you omitted Eternum on purpose :P (For context, I created and run Eternum, and that experience is mostly where my opinion of IPFS comes from.)

Gotcha! Thank you! Running a pinning service definitely still has rough edges =/ but I know the Infura team recently open sourced some of the tooling they built to make it a bit easier: https://blog.infura.io/new-tools-for-running-ipfs-nodes-196d.... Might help others who are self hosting a large chunk of data on a persistent node too...

If you ever want to chat about how we can make pinning services on IPFS easier to run, would love to chat! I know cluster has been researching how to improve "IPFS for enterprise" usage and would really appreciate the user feedback!

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#140

Post by lioeters »

knzhou wrote: post #130

Libgen is one of the greatest contributors to scientific productivity worldwide, possibly beaten only by Sci-Hub. Just about everybody in academia knows about it. If it ever vanished, some of us could probably still get by trading files from person to person, but nothing could be as perfect as what we got now.

> possibly beaten only by Sci-Hub

Today I learned that Library Genesis is actually "powered by Sci-Hub" as its primary source.

So I guess they're sister projects by similarly minded people (who seem to be mostly/originally based in Slavic countries, which I find interesting culturally - perhaps it's due to a looser legal environment + activist academics?).

> Just about everybody in academia knows about it.

That really says something about the state of society, this tension between copyright laws (and the motivations behind them) and the intellectual ideal of free and open access to knowledge.

Post reply on HN