Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Archivists Are Trying to Make Sure LibGen Never Goes Down

vice.com

Post reply on HN
Print view View original

161–170 of 270 posts

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#161

Post by izzydata »

burtonator wrote: post #71

What's interesting is that 32TB is becoming more and more affordable and the research material is roughly staying about the same size. That might change though as people start including video + data within papers and have new notebook formats that are live and contain docker containers/ipython, etc. It's a shame we can't just mail these around.

The bandwidth to upload to people can get expensive depending on where you live. Most home connections don't have bi-directional fiber so you are stuck with crippling amounts of upload bandwidth.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#162

Post by MadWombat »

miki123211 wrote: post #43

The new architecture of pirate sites, what I call the Hydra architecture, seems pretty interesting to me. There isn't a single site hosting the content, but a group of mirrors freely exchanging data between one another. In case some of them go down, the other ones still remain and new ones can appear, copying data from the remaining mirrors. This is like a hydra that grows two heads every time you chop one off. It's…

> It's absolutely unkillable

Just like any other distributed system, this is vulnerable to organized take downs and scare tactics. There was a whole bunch of mirrors of Pirate Bay, yet once most of Europe's legal systems adopted the "sharing is theft" mindset, it became pretty much impossible to find one.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#163

Post by gdy »

mikorym wrote: post #145
lioeters wrote: post #140

Earlier quoted context omitted.

> possibly beaten only by Sci-Hub Today I learned that Library Genesis is actually "powered by Sci-Hub" as its primary source. So I guess they're sister projects by similarly minded people (who seem to be mostly/originally based in Slavic countries, which I find interesting culturally - perhaps it's due to a looser legal environment + activist academics?). > Just about everybody in academia knows about it. That reall…

I am not an expert on the topic, but I believe that in the former Soviet Union it was common between mathematicians to pass around preprints (a la arXiv). These then perculated through to the West. I think it had to do with the USSR and their restrictive (if we are being euphemistic) policies towards academics.

"the USSR and their restrictive (if we are being euphemistic) policies towards academics."

What do you mean?

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#165

Post by frobozz »

EthanHeilman wrote: post #107

Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material becam…

> I do wonder wither digital or analogue formats are better able to survive into the distant future.

There are 5000 year old clay tablets we can still read.

There are centuries old documents on paper, vellum etc. that we can still read.

I personally have decades-old paper documents I can easily read, and a box of floppies I can't.

It's not just a problem of unreadable physical media, I have a database file on a perfectly readable HD that was generated by an application that is no longer available. I might be able to interrogate it somehow, but it won't be easy.

Digital formats and connectivity make LOCKSS easier, so that's a plus. There's less chance of a fire or flood or space-limited librarian destroying the last known copy. However, without archivists actively transforming content to new formats as required, it might only take a few decades before a lot of content starts to require a massive effort to read.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#166

Post by RiceAndWater »

lioeters wrote: post #140
knzhou wrote: post #130

Libgen is one of the greatest contributors to scientific productivity worldwide, possibly beaten only by Sci-Hub. Just about everybody in academia knows about it. If it ever vanished, some of us could probably still get by trading files from person to person, but nothing could be as perfect as what we got now.

> possibly beaten only by Sci-Hub Today I learned that Library Genesis is actually "powered by Sci-Hub" as its primary source. So I guess they're sister projects by similarly minded people (who seem to be mostly/originally based in Slavic countries, which I find interesting culturally - perhaps it's due to a looser legal environment + activist academics?). > Just about everybody in academia knows about it. That reall…

It's actually more the other way round, since Libgen stores all of sci-hub's papers for them.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#168

Post by lukebuehler »

To me, an aspiring scholar, LibGen is the most amazing tool ever. Things like inter-library loan and access to databases on university networks already make life so much easier to what it used to be—but nothing beats LibGen in terms of convenience. I’m in a the nowadays obscure field of patristic theology and I can’t believe how much stuff I can find on LibGen, often things that even highly specialized research libraries like Harvard don’t have.

The hours that LibGen saved me in gathering all the sources for my research must be in the hundreds. Thank you!

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#169

Post by EthanHeilman »

frobozz wrote: post #165
EthanHeilman wrote: post #107

Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material becam…

> I do wonder wither digital or analogue formats are better able to survive into the distant future. There are 5000 year old clay tablets we can still read. There are centuries old documents on paper, vellum etc. that we can still read. I personally have decades-old paper documents I can easily read, and a box of floppies I can't. It's not just a problem of unreadable physical media, I have a database file on a perfe…

Clay is the plastic of the ancient world.

Let's say the probability that: a single copy of a physical book survives 1,000 years, is found and is understood by an archaeologist, is pB and the probability that a single copy of a book on an SSD survives 1,000 years is found and understood by an archaeologist is pD. Even if pB is far larger than pD it could be the case that there might be so many more copies of single book held on SSDs thus making it more likely the book will survive via an SSD than a physical book. On the other hand the technology to recover data from SSDs might not exist in 1,000 years.

It could also be the case that each generation would copy these books onto new digital media providing an unbroken chain of copies. The oldest copy of the Iliad is Venetus.A which is from 1000AD (1000 years ago) despite the Iliad probably first being written down in 800BC (2800 years ago). It was copied from earlier copies of copies of copies.

I really don't know how this will play out and I've been unable to find research on how long SSD and flash memory based media survives especially if buried in a landfill.

* - If archaeologists exist in the future. The current push from the STEM boosters to defund and de-emphasize the humanities may result in a near-future without archaeologists or funded archaeological projects. Over 1,000 years the entire field could die.

Re: Archivists Are Trying to Make Sure LibGen Never Goes Down

#170

Post by asdff »

EthanHeilman wrote: post #169
frobozz wrote: post #165

Earlier quoted context omitted.

> I do wonder wither digital or analogue formats are better able to survive into the distant future. There are 5000 year old clay tablets we can still read. There are centuries old documents on paper, vellum etc. that we can still read. I personally have decades-old paper documents I can easily read, and a box of floppies I can't. It's not just a problem of unreadable physical media, I have a database file on a perfe…

Clay is the plastic of the ancient world. Let's say the probability that: a single copy of a physical book survives 1,000 years, is found and is understood by an archaeologist , is pB and the probability that a single copy of a book on an SSD survives 1,000 years is found and understood by an archaeologist is pD. Even if pB is far larger than pD it could be the case that there might be so many more copies of single b…

Would an SSD even function after 1000 years? Unless sealed, I imagine ambient moisture would do a number inside the drive. The same is true for books of course, but we still have 1000 year old books that have lasted by sitting on a shelf in churches and temples, etc., without any specific care until recent history.

The nice part of a book in an apocalyptic scenario is that you can copy it even if you don't know the language. You don't need a special tool for this, only one capable of marking a surface. It wouldn't be fun or fast, but it's possible and it's what monks did for centuries. Would archeologists 1000 years from now be lucky enough to find a SATA cable too?

Post reply on HN