51–60 of 156 posts
Related question, has Anna's archive been thoroughly filtered for non-copyright-related illegal material? Pedo, terrorism, etc. I've considered downloading a few chunks of it but I'm worried of ending up with content I really don't want to be anywhere near from.
Earlier quoted context omitted.
It would be incredible for LLMs. Searching it, using it as training data, etc. Would probably have to be done in Russia or some other country that doesn't respect international copyright though.
LLMs already use it, dude )
Honestly I don't think it would be that costly, but it would take a pretty long time to put together. I have a (few years old) copy of Library Genesis converted to plaintext and it's around 1TB. I think libgen proper was 50-100TB at the time, so we can probably assume that AA (~1PB) would be around 10-20TB when converted to plaintext. You'd probably spend several weeks torrenting a chunk of the archive, converting ev…
I think there’s a couple ways to improve it: 1. There’s a lot of variants of the same book. We only need one for the index. Perhaps for each ISBN, select the format easiest to parse. 2. We can download, convert and index top 100K books first, launch with these, and then continue indexing and adding other books.
I count 80+ editions of J.R.R. Tolkien's _The Hobbit_ at:
https://tolkienlibrary.com/booksbytolkien/hobbit/editions.ph...
granted some predate ISBNs, one is the 3D pop-up version, so not a traditional text, and so forth, but filtering by ISBN will _not_ filter out duplicates.
There is also the problem of the same work being published under multiple titles (and also ISBNs) --- Hal Clement's _Small Changes_ was re-published as _Space Lash_ and that short story collection is now collected in:
https://www.goodreads.com/book/show/939760.Music_of_Many_Sph...
along with others.
Earlier quoted context omitted.
Considering the anarchist cookbook is just a rebranded selection of freely-available US Army Field Manuals, ... I don't see the problem.
I don't either, but many states have laws regarding books on how to build bombs and they might get enforced more than copyright.
https://news.sky.com/story/anarchist-cookbook-case-student-j...
Earlier quoted context omitted.
> It's trivial to normalise the various formats, Ha. Ha. ha ha ha. As someone who as pretty broadly tried to normalize a pile of books and documents I have legitimate access to, no it is not . You can get good results 80% of the time, usable but messy results 18% of the time, and complete garbage the remaining 2%. More effort seems to only result in marginal improvements.
98% sounds good enough for the usecase suggested here.
Edit: my point is that I would like to share my work but that is hard to do in a legal way. That is the main reason I gave up.
Earlier quoted context omitted.
There should be a way to leverage compression when storing multiple editions of the same book.
From a good search perspective though you probably dont want 500 different versions of the same book popping up for a query
As far as I know, no one has fully implemented full-text search directly over Anna's Archive. Technically it’s feasible with tools like Meilisearch, Elasticsearch, or Lucene, but the main challenges are: Converting all documents (PDFs, EPUBs, etc.) to clean plaintext. Indexing at scale efficiently. Managing potential legal issues. Z-Library does something similar, but it’s smaller in scope and doesn't integrate AA’s…
Honestly I don't think it would be that costly, but it would take a pretty long time to put together. I have a (few years old) copy of Library Genesis converted to plaintext and it's around 1TB. I think libgen proper was 50-100TB at the time, so we can probably assume that AA (~1PB) would be around 10-20TB when converted to plaintext. You'd probably spend several weeks torrenting a chunk of the archive, converting ev…
I think there’s a couple ways to improve it: 1. There’s a lot of variants of the same book. We only need one for the index. Perhaps for each ISBN, select the format easiest to parse. 2. We can download, convert and index top 100K books first, launch with these, and then continue indexing and adding other books.
Earlier quoted context omitted.
I think there’s a couple ways to improve it: 1. There’s a lot of variants of the same book. We only need one for the index. Perhaps for each ISBN, select the format easiest to parse. 2. We can download, convert and index top 100K books first, launch with these, and then continue indexing and adding other books.
There should be a way to leverage compression when storing multiple editions of the same book.