Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has anybody built search on top of Anna's Archive?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 156 posts

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#111

Post by 1vuio0pswjnm7 »

serial_dev wrote: post #15
bhaney wrote: post #3

Honestly I don't think it would be that costly, but it would take a pretty long time to put together. I have a (few years old) copy of Library Genesis converted to plaintext and it's around 1TB. I think libgen proper was 50-100TB at the time, so we can probably assume that AA (~1PB) would be around 10-20TB when converted to plaintext. You'd probably spend several weeks torrenting a chunk of the archive, converting ev…

The main barriers for me would be: 1. Why? Who would use that? What’s the problem with the other search engines? How will it be paid for? 2. Potential legal issues. The technical barriers are at least challenging and interesting. Providing a service with significant upfront investment needs with no product or service vision that I’ll likely to be sued for a couple of times a year, probably losing with who knows what…

But he did not mention anything about creating a "service"

It could be his own copy for personal use

What if computers continue to become faster and storage continues to become cheaper; what if "large" amounts data continue to become more manageable

The data might seem large today, but it might not seem large or unmanageable in the future

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#112

Post by gosub100 »

bilekas wrote: post #21
simgt wrote: post #14

Related question, has Anna's archive been thoroughly filtered for non-copyright-related illegal material? Pedo, terrorism, etc. I've considered downloading a few chunks of it but I'm worried of ending up with content I really don't want to be anywhere near from.

This is a really strange question to be honest you could ask this literally about any download let alone simply torrents of documents.

It's the textbook example of the "chilling effect" created by mass surveillance.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#113

Post by lukan »

dns_snek wrote: post #51
simgt wrote: post #14

Related question, has Anna's archive been thoroughly filtered for non-copyright-related illegal material? Pedo, terrorism, etc. I've considered downloading a few chunks of it but I'm worried of ending up with content I really don't want to be anywhere near from.

Download everything, we know that laws don't apply when you do it on a large enough scale. Not legal advice.

I think you got that wrong. Laws only don't apply if you are large enough. (Like Meta)

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#114

Post by rthnbgrredf »

moffkalast wrote: post #82
trollbridge wrote: post #76

Earlier quoted context omitted.

Decent storage is $10/TB, so for $10,000 you could just keep the entire 1PB of data. A rather obvious question is if someone has trained an LLM on this archive yet.

A rather obvious answer is Meta is currently being sued for training Llama on Anna's archive. You can be practically certain that every notable LLM has been trained on it.

> You can be practically certain that every notable LLM has been trained on it.

But only Meta was kind of not so smart to publicly admit it.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#116

Post by coolThingsFirst »

bbor wrote: post #81
serial_dev wrote: post #15

Earlier quoted context omitted.

The main barriers for me would be: 1. Why? Who would use that? What’s the problem with the other search engines? How will it be paid for? 2. Potential legal issues. The technical barriers are at least challenging and interesting. Providing a service with significant upfront investment needs with no product or service vision that I’ll likely to be sued for a couple of times a year, probably losing with who knows what…

1. It'd be for the scientific community (broadly-construed). Converting media that is currently completely un-indexed into plaintext and offering a suite of search features for finding content within it would be a game-changer, IMO! If you've ever done a lit review for any field other than ML, I'm guessing you know how reliant many fields are on relatively-old books and articles (read: PDFs at best, paper-only at wor…

Yeah but how does the search work, does it show a portion of the text? If it's a portion of the text isn't that also a part of the book?

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#117

Post by perdomon »

net01 wrote: post #50

They did! They conducted a competition https://annas-archive.org/blog/all-isbns-winners.html , in which a few submissions exceeded the minimum requirements and implemented a good search tool & visualiser.

I think OP was more interested in the ability to text search through the contents. This competition was great and some of the entries were really informative, but none of them included a full text search of the contents of all books.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#118

Post by gosub100 »

lukan wrote: post #29
DocTomoe wrote: post #26

Earlier quoted context omitted.

Considering the anarchist cookbook is just a rebranded selection of freely-available US Army Field Manuals, ... I don't see the problem.

I don't either, but many states have laws regarding books on how to build bombs and they might get enforced more than copyright.

Not saying you're deceiving but can you show me where a state has made a book about bombs illegal? It seems like that would be a slam dunk 1A violation. And yes I'm aware that states willfully violate 2A but I don't want to discuss it here.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#119

Post by jmb99 »

podgorniy wrote: post #40

There is a search solution for zipped fb2 files. Not exactly what you need, but it has potential. The project has similar story to Anna's archive. There is 0.5 TB of archived books, and the project creates index of all the books with text, title and aruthor search capabilities, gives html UI for search and reading. On weak machine it takes about 2 hours to build that index. So if you have zipped archives of fb2, you…

Worth mentioning that 0.5TB is tiny compared to Anna’s, which currently sits around 1.1PB.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#120

Post by wolfgang42 »

showerst wrote: post #63
ThatPlayer wrote: post #48

Earlier quoted context omitted.

I've done something similar with a static hosted site I'm working on. I opted to not reinvent the wheel, and just use WASM Sqlite in the browser. Sqlite already splits the database into fixed-size pages, so the driver using HTTP Range Requests can download only the required pages. Just have to make good indexes. I can even use Sqlite's full-text search capabilities!

How would that scale to 10TB+ of plain text though? Presumably the indexes would be many gigabytes, especially with full text search.

The client only needs to get indexes for the specific search; if the index is just a list of TF-IDF term scores per document (which gets you a very reasonable start on search relevance) some extremely back-of-the-envelope math leads me to guess at an upper bound in the low tens of megabytes per (non-stopword) term, which seems doable for a client to download on demand.
Post reply on HN