Ask HN: Has anybody built search on top of Anna's Archive? • Hacker News Board

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#81

Post by bbor » Wed, Jun 04, 2025, 1:58 PM UTC

serial_dev wrote: post #15

bhaney wrote: post #3

Honestly I don't think it would be that costly, but it would take a pretty long time to put together. I have a (few years old) copy of Library Genesis converted to plaintext and it's around 1TB. I think libgen proper was 50-100TB at the time, so we can probably assume that AA (~1PB) would be around 10-20TB when converted to plaintext. You'd probably spend several weeks torrenting a chunk of the archive, converting ev…

The main barriers for me would be: 1. Why? Who would use that? What’s the problem with the other search engines? How will it be paid for? 2. Potential legal issues. The technical barriers are at least challenging and interesting. Providing a service with significant upfront investment needs with no product or service vision that I’ll likely to be sued for a couple of times a year, probably losing with who knows what…

1. It'd be for the scientific community (broadly-construed). Converting media that is currently completely un-indexed into plaintext and offering a suite of search features for finding content within it would be a game-changer, IMO! If you've ever done a lit review for any field other than ML, I'm guessing you know how reliant many fields are on relatively-old books and articles (read: PDFs at best, paper-only at worst) that you can basically only encounter via a) citation chains, b) following an author, or c) encyclopedias/textbooks.

2. I really don't see how this could ever lead to any kind of legal issue. You're not hosting any of the content itself, just offering a search feature for it. GoodReads doesn't need legal permission to index popular books, for example.

In general I get the sense that your comment is written from the perspective of an entrepreneur/startup mindset. I'm sure that's brought you meaning and maybe even some wealth, but it's not a universal one! Some of us are more interested in making something to advance humanity than something likely to make a profit, even if we might look silly in the process.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#82

Post by moffkalast » Wed, Jun 04, 2025, 2:01 PM UTC

trollbridge wrote: post #76

bhaney wrote: post #3

Honestly I don't think it would be that costly, but it would take a pretty long time to put together. I have a (few years old) copy of Library Genesis converted to plaintext and it's around 1TB. I think libgen proper was 50-100TB at the time, so we can probably assume that AA (~1PB) would be around 10-20TB when converted to plaintext. You'd probably spend several weeks torrenting a chunk of the archive, converting ev…

Decent storage is $10/TB, so for $10,000 you could just keep the entire 1PB of data. A rather obvious question is if someone has trained an LLM on this archive yet.

A rather obvious answer is Meta is currently being sued for training Llama on Anna's archive.

You can be practically certain that every notable LLM has been trained on it.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#83

Post by maartin0 » Wed, Jun 04, 2025, 2:14 PM UTC

teekert wrote: post #47

Probably this was already done at Google, Meta, X and OpenAI, before training their LLMs.

There's actually section in the Wikipedia page that explicitly says DeepSeek was trained on it

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#84

Post by qcic » Wed, Jun 04, 2025, 2:16 PM UTC

ThatPlayer wrote: post #48

tomthe wrote: post #16

Earlier quoted context omitted.

I wonder if you could implement it with only static hosting? We would need to split the index into a lot of smaller files that can be practically downloaded by browsers, maybe 20 MB each. The user types in a search query, the browser hashes the query and downloads the corresponding index file which contains only results for that hashed query. Then the browser sifts quickly through that file and gives you the result.…

I've done something similar with a static hosted site I'm working on. I opted to not reinvent the wheel, and just use WASM Sqlite in the browser. Sqlite already splits the database into fixed-size pages, so the driver using HTTP Range Requests can download only the required pages. Just have to make good indexes. I can even use Sqlite's full-text search capabilities!

Super interesting.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#85

Post by gpm » Wed, Jun 04, 2025, 2:30 PM UTC

HDThoreaun wrote: post #61

IlikeKitties wrote: post #41

Earlier quoted context omitted.

In fact, facebook torrented annas archive and got busted for it, because of course they did: https://torrentfreak.com/meta-torrented-over-81-tb-of-data-t...

Every LLM maker probably did the same. Facebook just has disgruntled employees who leaked it

Google goes around legally scanning every book they can get their hands on with books.google.com. Legally scanning every paper they can get their hands on with scholar.google.com.

I doubt they'd resort to piracy for what is basically the same information as what they've already legally acquired...

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#86

Post by Aachen » Wed, Jun 04, 2025, 2:33 PM UTC

bbor wrote: post #81

serial_dev wrote: post #15

Earlier quoted context omitted.

The main barriers for me would be: 1. Why? Who would use that? What’s the problem with the other search engines? How will it be paid for? 2. Potential legal issues. The technical barriers are at least challenging and interesting. Providing a service with significant upfront investment needs with no product or service vision that I’ll likely to be sued for a couple of times a year, probably losing with who knows what…

1. It'd be for the scientific community (broadly-construed). Converting media that is currently completely un-indexed into plaintext and offering a suite of search features for finding content within it would be a game-changer, IMO! If you've ever done a lit review for any field other than ML, I'm guessing you know how reliant many fields are on relatively-old books and articles (read: PDFs at best, paper-only at wor…

> I really don't see how this could ever lead to any kind of legal issue. You're not hosting any of the content itself, just offering a search feature for it.

You don't need to host copyrighted material. It's all about intent. The Pirate Bay is (imo correctly, even if I disagree with other aspects about copyright law and its enforcement) seen as a place where people go to find ways to not pay authors for their content. They never hosted a copyrighted byte but they're banned in some form (DNS, IP, domain seizures) in many countries. Proxies of TPB also, so being like an ISP for such a site is already enough, whereas nobody is ordering blocks of Comcast's IP addresses for providing access to websites with copyrighted material because they didn't have a somewhat-provable intent to provide copyright infringement

When I read the OP, I imagine this would link from the search results directly to Anna's archive and sci-hub, but I think you'd have to spin it as a general purpose search page and ideally not even mention AA was one of the sources, much less have links

(Don't get me wrong: everyone wants this except the lobby of journals that presently own the rights)

It would be a real shame if an anonymous third party that's definitely not the website operator made a Firefox add-on that illegitimately inserts these links to search results page though

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#87

Post by calibas » Wed, Jun 04, 2025, 2:34 PM UTC

1970-01-01 wrote: post #77

bethekidyouwant wrote: post #72

Earlier quoted context omitted.

It wouldn’t be called full text search of AA, It would be called full tech search of every book in the world.

You are asking a judge to consider that a book is ok to scrape because it's part of a much larger collection of books, perhaps the biggest and best collection, and therefore it's all OK because at scale means good.

Google already successfully argued in court that creating an online search index of books constitutes fair use: https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,....

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#88

Post by [deleted] » Wed, Jun 04, 2025, 2:34 PM UTC

andrepd wrote: post #70

namlem wrote: post #28

Earlier quoted context omitted.

It would be incredible for LLMs. Searching it, using it as training data, etc. Would probably have to be done in Russia or some other country that doesn't respect international copyright though.

> Would probably have to be done in Russia or some other country that doesn't respect international copyright though. Incredible, several years of major American AI companies showing that flaunting copyright only matters if it's college kids torrenting shows or enthusiasts archiving bootlegs on whatcd, but if it's big corpos doing it it's necessary for innovation. Yet some people still believe "it would have to be do…

[deleted]

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#89

Post by corgi912 » Wed, Jun 04, 2025, 2:39 PM UTC

TeMPOraL wrote: post #62

executesorder66 wrote: post #42

Earlier quoted context omitted.

> or some other country that doesn't respect international copyright though. Like the US? OpenAI et al. don't give a shit.

There's a difference between feeding massive amounts of copyrighted material to a training process that blends them thoroughly and irreversibly, and doing all that in-house, vs. offering people a service that indexes (and possibly partially rehosts) that material, enabling and encouraging users to engage directly in pirating concrete copyrighted works.

There's this famous phrase in Russian that was born out of a short interview with a woman, a strong Putin supporter, that's often been used as a sarcastic remark for pointing out someone's double standards and/or hypocrisy.

It can be roughly translated to "you don't understand, it's a completely different situation". That's what's constantly on my mind when I'm reading discussions like this one.

Everybody and their dog torrenting petabytes of data and getting away with it (Meta is the only one that got caught and they've still gotten away with doing it)?

The very same data poor American students were forced to commit suicide over? The same data that average American housewives were sued over for millions of dollars of "damages"? The same data that often gets random German plumbers or steelworkers to pay thousands of euros of "fines" to the copyright mafia so they won't get sued and have their lives ruined?

Yet when giant corporations are doing the exact same thing on a massive scale, it's fine? It's not even the same thing, an American student torrenting books isn't making any money off it, while Meta very much is.

Of course it's not the same, a simple-minded and poorly educated person like me isn't capable of understanding the difference. You keep believing in your moral superiority, the rest of the world has finally woken up.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#90

Post by DataDaoDe » Wed, Jun 04, 2025, 2:39 PM UTC

andrepd wrote: post #70

namlem wrote: post #28

Earlier quoted context omitted.

It would be incredible for LLMs. Searching it, using it as training data, etc. Would probably have to be done in Russia or some other country that doesn't respect international copyright though.

> Would probably have to be done in Russia or some other country that doesn't respect international copyright though. Incredible, several years of major American AI companies showing that flaunting copyright only matters if it's college kids torrenting shows or enthusiasts archiving bootlegs on whatcd, but if it's big corpos doing it it's necessary for innovation. Yet some people still believe "it would have to be do…

OP does have an exaggerated statement - its not like there aren't laws in Russia or something and I largely agree with your sentiment. I think there are levels to this though and its pretty clear that Russia is much riskier than the USA when it comes to IP - just look up anything to do with insuring IP risk in Russia (here's one such example: https://baa.no/en/articles/i-have-ip-in-russia-is-my-ip-at-r...)

Also according to the office of US trade representative, Russia is on the priority watch list of countries that do not respect IP [1] and post 2022, largely due to the war, Russia implemented measures negatively effecting IP rights. [2,3]

If you think it isn't the case and Russia is just as risky as the US when it comes to copyright and IP, I would be interested to know why.

1. https://ustr.gov/about/policy-offices/press-office/press-rel... 2. https://www.papula-nevinpat.com/executive-summary-the-ip-sit... 3. https://www.taftlaw.com/news-events/law-bulletins/russia-iss...