Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has anybody built search on top of Anna's Archive?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 156 posts

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#41

Post by IlikeKitties »

jxjnskkzxxhx wrote: post #35
namlem wrote: post #28

Earlier quoted context omitted.

It would be incredible for LLMs. Searching it, using it as training data, etc. Would probably have to be done in Russia or some other country that doesn't respect international copyright though.

Do you have a reason to believe this ain't already being done? I would assume that the big guys like openai are already training on basically all text in existence.

In fact, facebook torrented annas archive and got busted for it, because of course they did:

https://torrentfreak.com/meta-torrented-over-81-tb-of-data-t...

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#42

Post by executesorder66 »

namlem wrote: post #28
serial_dev wrote: post #15

Earlier quoted context omitted.

The main barriers for me would be: 1. Why? Who would use that? What’s the problem with the other search engines? How will it be paid for? 2. Potential legal issues. The technical barriers are at least challenging and interesting. Providing a service with significant upfront investment needs with no product or service vision that I’ll likely to be sued for a couple of times a year, probably losing with who knows what…

It would be incredible for LLMs. Searching it, using it as training data, etc. Would probably have to be done in Russia or some other country that doesn't respect international copyright though.

> or some other country that doesn't respect international copyright though.

Like the US? OpenAI et al. don't give a shit.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#43

Post by tangus »

podgorniy wrote: post #40

There is a search solution for zipped fb2 files. Not exactly what you need, but it has potential. The project has similar story to Anna's archive. There is 0.5 TB of archived books, and the project creates index of all the books with text, title and aruthor search capabilities, gives html UI for search and reading. On weak machine it takes about 2 hours to build that index. So if you have zipped archives of fb2, you…

But fb2 files are marked up text, which is (relatively) trivial to index. The bulk of Anna's Archive's books are made of from scanned images.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#48

Post by ThatPlayer »

tomthe wrote: post #16
bhaney wrote: post #3

Honestly I don't think it would be that costly, but it would take a pretty long time to put together. I have a (few years old) copy of Library Genesis converted to plaintext and it's around 1TB. I think libgen proper was 50-100TB at the time, so we can probably assume that AA (~1PB) would be around 10-20TB when converted to plaintext. You'd probably spend several weeks torrenting a chunk of the archive, converting ev…

I wonder if you could implement it with only static hosting? We would need to split the index into a lot of smaller files that can be practically downloaded by browsers, maybe 20 MB each. The user types in a search query, the browser hashes the query and downloads the corresponding index file which contains only results for that hashed query. Then the browser sifts quickly through that file and gives you the result.…

I've done something similar with a static hosted site I'm working on. I opted to not reinvent the wheel, and just use WASM Sqlite in the browser. Sqlite already splits the database into fixed-size pages, so the driver using HTTP Range Requests can download only the required pages. Just have to make good indexes.

I can even use Sqlite's full-text search capabilities!

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#49

Post by throwup238 »

notpushkin wrote: post #7
bhaney wrote: post #3

Honestly I don't think it would be that costly, but it would take a pretty long time to put together. I have a (few years old) copy of Library Genesis converted to plaintext and it's around 1TB. I think libgen proper was 50-100TB at the time, so we can probably assume that AA (~1PB) would be around 10-20TB when converted to plaintext. You'd probably spend several weeks torrenting a chunk of the archive, converting ev…

I think there’s a couple ways to improve it: 1. There’s a lot of variants of the same book. We only need one for the index. Perhaps for each ISBN, select the format easiest to parse. 2. We can download, convert and index top 100K books first, launch with these, and then continue indexing and adding other books.

How are you going to download the top 100k? The only reasonable way to download that many books from AA or Libgen is to use the torrents, which are sorted sequentially by upload date.

I tried to automate downloading just a thousand books and it was unbearably slow, from IPFS or the mirrors both. I ended up picking the individual files out of the torrents. Even just identifying or deduping the top 100k would be a significant task.

Post reply on HN