Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has anybody built search on top of Anna's Archive?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 156 posts

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#31

Post by lukan »

bilekas wrote: post #27
lukan wrote: post #20

Earlier quoted context omitted.

He said he wants to download lots of it in general, not specifical. Legit question, if you end up with dark material. I would assume pedo stuff is not really there, but the anarchist cookbook and alike likely will be.

I'm still not sure the question makes much sense, if it's a general: "I want to support the project and so I want to seed a large chunk" Okay, I guess it's your due diligence to check, but there is a reporting feature built in, if something is found, report it. Aside from that, if you're searching for specific content, the question is moot I guess. I guess my confusion is what distinguishes this apart from any other…

I understood it as he or she wants to download large chunks of potentially interesting books for offline use, or once Anna goes down. So a broad filter. Not for seeding.

But thanks for the explanation that there is a report build in.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#33

Post by lukan »

oguz-ismail wrote: post #23
lukan wrote: post #20

Earlier quoted context omitted.

He said he wants to download lots of it in general, not specifical. Legit question, if you end up with dark material. I would assume pedo stuff is not really there, but the anarchist cookbook and alike likely will be.

>I would assume pedo stuff is not really there Search for "lolicon"

Well, I won't. But does it contain just text or real pictures? That would make a big legal difference I assume.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#34

Post by sam_lowry_ »

namlem wrote: post #28
serial_dev wrote: post #15

Earlier quoted context omitted.

The main barriers for me would be: 1. Why? Who would use that? What’s the problem with the other search engines? How will it be paid for? 2. Potential legal issues. The technical barriers are at least challenging and interesting. Providing a service with significant upfront investment needs with no product or service vision that I’ll likely to be sued for a couple of times a year, probably losing with who knows what…

It would be incredible for LLMs. Searching it, using it as training data, etc. Would probably have to be done in Russia or some other country that doesn't respect international copyright though.

LLMs already use it, dude )

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#35

Post by jxjnskkzxxhx »

namlem wrote: post #28
serial_dev wrote: post #15

Earlier quoted context omitted.

The main barriers for me would be: 1. Why? Who would use that? What’s the problem with the other search engines? How will it be paid for? 2. Potential legal issues. The technical barriers are at least challenging and interesting. Providing a service with significant upfront investment needs with no product or service vision that I’ll likely to be sued for a couple of times a year, probably losing with who knows what…

It would be incredible for LLMs. Searching it, using it as training data, etc. Would probably have to be done in Russia or some other country that doesn't respect international copyright though.

Do you have a reason to believe this ain't already being done? I would assume that the big guys like openai are already training on basically all text in existence.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#36

Post by sam_lowry_ »

bendangelo wrote: post #10
imdavidsantiago wrote: post #8

As far as I know, no one has fully implemented full-text search directly over Anna's Archive. Technically it’s feasible with tools like Meilisearch, Elasticsearch, or Lucene, but the main challenges are: Converting all documents (PDFs, EPUBs, etc.) to clean plaintext. Indexing at scale efficiently. Managing potential legal issues. Z-Library does something similar, but it’s smaller in scope and doesn't integrate AA’s…

I’ve done something like this before. Meilisearch will not be viable, because it indexes very slow and it takes up a lot of space. In my experience only Tantivy can index this much data. Check out Lnx.

Lucene would fo fine as well, I guess. As much as I like the author of Tantivy, it is a toy compared to Lucene.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#37

Post by jxjnskkzxxhx »

oguz-ismail wrote: post #23
lukan wrote: post #20

Earlier quoted context omitted.

He said he wants to download lots of it in general, not specifical. Legit question, if you end up with dark material. I would assume pedo stuff is not really there, but the anarchist cookbook and alike likely will be.

>I would assume pedo stuff is not really there Search for "lolicon"

I thought that was anime pictures...?

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#38

Post by bawolff »

palmfacehn wrote: post #12
notpushkin wrote: post #7

Earlier quoted context omitted.

I think there’s a couple ways to improve it: 1. There’s a lot of variants of the same book. We only need one for the index. Perhaps for each ISBN, select the format easiest to parse. 2. We can download, convert and index top 100K books first, launch with these, and then continue indexing and adding other books.

There should be a way to leverage compression when storing multiple editions of the same book.

From a good search perspective though you probably dont want 500 different versions of the same book popping up for a query

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#39

Post by bawolff »

fake-name wrote: post #25
greggsy wrote: post #17

Earlier quoted context omitted.

It's trivial to normalise the various formats, and there were a few libraries and ML models to help parse PDFs. I was tinkering around with something like this for academic papers in Zotero, and the main issue I ran into was words spilling over to the next page, and footnotes. I totally gave up on that endeavour several years ago, but the tooling has probably matured exponentially since then. As an example, all the a…

> It's trivial to normalise the various formats, Ha. Ha. ha ha ha. As someone who as pretty broadly tried to normalize a pile of books and documents I have legitimate access to, no it is not . You can get good results 80% of the time, usable but messy results 18% of the time, and complete garbage the remaining 2%. More effort seems to only result in marginal improvements.

98% sounds good enough for the usecase suggested here.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#40

Post by podgorniy »

There is a search solution for zipped fb2 files. Not exactly what you need, but it has potential.

The project has similar story to Anna's archive. There is 0.5 TB of archived books, and the project creates index of all the books with text, title and aruthor search capabilities, gives html UI for search and reading. On weak machine it takes about 2 hours to build that index.

So if you have zipped archives of fb2, you can use the project to create web UI with search for those files. Without need of enough space to unpack all the files.

You'll have to translate some russian though to get instructions on how to set it up.

https://gitlab.com/opennota/fb2index/-/blob/master/README.ru...

Post reply on HN