Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has anybody built search on top of Anna's Archive?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 156 posts

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#141

Post by lukan »

gosub100 wrote: post #131
lukan wrote: post #127

Earlier quoted context omitted.

Not saying you cannot read, but if you would, the other answer to my comment literally has such an example. Germany is like this as well since a few years. Not all states are within the US.

you're not saying I cannot read, but that type of inflection is uncalled for. You have been reported to the mods.

Uncalled for?

"Not saying you're deceiving"

Right next to your answer were you implied I might be deceiving was already an answer telling apparently I was not deceiving. So yes, the mocking of my comment was not up to HN standards, but you don't see how you started it?

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#142

Post by TeMPOraL »

sneak wrote: post #136
corgi912 wrote: post #89

Earlier quoted context omitted.

There's this famous phrase in Russian that was born out of a short interview with a woman, a strong Putin supporter, that's often been used as a sarcastic remark for pointing out someone's double standards and/or hypocrisy. It can be roughly translated to "you don't understand, it's a completely different situation". That's what's constantly on my mind when I'm reading discussions like this one. Everybody and their d…

> The very same data poor American students were forced to commit suicide over Leaving the rest of your argument aside, precisely nobody forced aaronsw to commit suicide.

There's also a matter of 'aaronsw being a student, not many "poor American students" as GP implies. As far as I know, this was the only case of this type[0][1].

Honestly was too tired to point that out in my earlier reply, but that's exactly the kind of argument you get when people are not willing (or purposefully refusing) to consider details. Intentionally or not, you get bogus and highly manipulative statements.

A single case of a student activist fighting for freedom of communication and access to public goods for citizens, ending up breaking under pressure from public/non-profit institutions MIT, JSTOR, FBI over copyright, is not the same as what GP implied - many students, regular folks just like you and me, being forced to take their own lives due to legal consequences of pirating books in bulk. Nothing like the latter ever happened anyway.

We can do better than this.

(And even if we can't, I trust the courts can.)

--

[0] - Curiously, while doing some search now to be sure I didn't miss any similar case, I learned that JSTOR incident wasn't the first for 'aaronsw - apparently, he did the same thing a few years earlier with public court documents[1]; FBI investigated this too, and concluded he was legally in the clear. It's probably well-known to everyone here, but I somehow missed it, so #TodayILearned.

[1] - https://en.wikipedia.org/wiki/Aaron_Swartz#PACER

[2] - https://en.wikipedia.org/wiki/Edwin_Howard_Armstrong was the only one I could find that was even remotely related - an engineer and inventor who, in big part due to prolonged fighting over patents consuming all his time and money, suffered from a mental breakdown and committed suicide at 63.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#143

Post by notpushkin »

throwup238 wrote: post #49
notpushkin wrote: post #7

Earlier quoted context omitted.

I think there’s a couple ways to improve it: 1. There’s a lot of variants of the same book. We only need one for the index. Perhaps for each ISBN, select the format easiest to parse. 2. We can download, convert and index top 100K books first, launch with these, and then continue indexing and adding other books.

How are you going to download the top 100k? The only reasonable way to download that many books from AA or Libgen is to use the torrents, which are sorted sequentially by upload date. I tried to automate downloading just a thousand books and it was unbearably slow, from IPFS or the mirrors both. I ended up picking the individual files out of the torrents. Even just identifying or deduping the top 100k would be a sign…

For each book they store its exact location in the torrent files. You can see on the book page, e.g.:

collection “ia” → torrent “annas-archive-ia-acsm-n.tar.torrent” → file “annas-archive-ia-acsm-n.tar” (extract) → file “notesonsynthesis0000unse.pdf”

But probably you should get it from the database dumps they provide instead of hammering the website.

So you come up with a list of books you want to prioritize, search the DB for torrent name and file to download, download only the files you need, and extract them. You’ll probably end up with quite a few more books, which you may index or skip for now, but it is certainly doable.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#144

Post by notpushkin »

WillAdams wrote: post #53
notpushkin wrote: post #7

Earlier quoted context omitted.

I think there’s a couple ways to improve it: 1. There’s a lot of variants of the same book. We only need one for the index. Perhaps for each ISBN, select the format easiest to parse. 2. We can download, convert and index top 100K books first, launch with these, and then continue indexing and adding other books.

The thing is, for an ISBN, that is one edition, by one publisher and one can easily have the same text under 3 different ISBNs from one publisher (hardcover, trade paperback, mass-market paperback). I count 80+ editions of J.R.R. Tolkien's _The Hobbit_ at: https://tolkienlibrary.com/booksbytolkien/hobbit/editions.ph... granted some predate ISBNs, one is the 3D pop-up version, so not a traditional text, and so forth,…

Hmmm, yeah, ISBN isn’t great for this. Is there a good way to deduplicate the books by their contents?

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#145

Post by notpushkin »

qingcharles wrote: post #126
bawolff wrote: post #38

Earlier quoted context omitted.

From a good search perspective though you probably dont want 500 different versions of the same book popping up for a query

And without some sort of weighting system, it wouldn't even know which one is the best one to show the user.

We’ll also need to consider that some versions might be easier to index even though the user would prefer another version. E.g. if we have a TXT and EPub, we might want to index TXT (if it’s clean enough), but present user with EPub (with formatting and stuff).

But it’s not a huge problem actually: just link to the search page instead and let the user decide what they want to download.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#146

Post by WillAdams »

notpushkin wrote: post #144
WillAdams wrote: post #53

Earlier quoted context omitted.

The thing is, for an ISBN, that is one edition, by one publisher and one can easily have the same text under 3 different ISBNs from one publisher (hardcover, trade paperback, mass-market paperback). I count 80+ editions of J.R.R. Tolkien's _The Hobbit_ at: https://tolkienlibrary.com/booksbytolkien/hobbit/editions.ph... granted some predate ISBNs, one is the 3D pop-up version, so not a traditional text, and so forth,…

Hmmm, yeah, ISBN isn’t great for this. Is there a good way to deduplicate the books by their contents?

LoC or Dewey Decimal with author and title (and edition?) should work.

I wish there was some better book cataloging/organizing scheme --- the Online Books Page uses LoC:

https://onlinebooks.library.upenn.edu/subjects.html

and is the most workable of the indices I've used.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#147

Post by tomhow »

lukan wrote: post #141
gosub100 wrote: post #131

Earlier quoted context omitted.

you're not saying I cannot read, but that type of inflection is uncalled for. You have been reported to the mods.

Uncalled for? "Not saying you're deceiving" Right next to your answer were you implied I might be deceiving was already an answer telling apparently I was not deceiving. So yes, the mocking of my comment was not up to HN standards, but you don't see how you started it?

I think you're each setting the other off and being a bit overreactive to each other's comments, and I think there may be a misunderstanding of the other's intent. We still need to make an effort to observe the guidelines even if a reply to us comment seems hostile. Sometimes it's best to just to stop.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#148

Post by carlosjobim »

DaSHacka wrote: post #101
Aachen wrote: post #86

Earlier quoted context omitted.

> I really don't see how this could ever lead to any kind of legal issue. You're not hosting any of the content itself, just offering a search feature for it. You don't need to host copyrighted material. It's all about intent . The Pirate Bay is (imo correctly, even if I disagree with other aspects about copyright law and its enforcement) seen as a place where people go to find ways to not pay authors for their conte…

> When I read the OP, I imagine this would link from the search results directly to Anna's archive and sci-hub You could just give users ISBNs or link to the book's metadata on openlibrary[0], both of which AA's native search already does. [0] https://openlibrary.org/

Exactly.

1. The ISBN in cleartext

2. An isbn://123123123 link

3. A link to the book on a legal library borrowing service

4. A link to buy the book on Amazon

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#149

Post by lukan »

tomhow wrote: post #147
lukan wrote: post #141

Earlier quoted context omitted.

Uncalled for? "Not saying you're deceiving" Right next to your answer were you implied I might be deceiving was already an answer telling apparently I was not deceiving. So yes, the mocking of my comment was not up to HN standards, but you don't see how you started it?

I think you're each setting the other off and being a bit overreactive to each other's comments, and I think there may be a misunderstanding of the other's intent. We still need to make an effort to observe the guidelines even if a reply to us comment seems hostile. Sometimes it's best to just to stop.

Indeed. Actually I did not try to attack. My intention was showing mild irony to something I perceived as an attack.

Those nuances easily get lost in text, I know, but that my post got flagged and his initial one did not, I really did not liked and that angered me a bit. But I can live with that, without making a drama out of it. Thanks for trying to mediate.

Re: Ask HN: Has anybody built search on top of Anna's Archive?

#150

Post by carlosjobim »

fazlerocks wrote: post #138

The indexing costs would be nuts - Anna's Archive is like 200TB+ and growing fast. Even with decent search infra you're looking at serious compute/storage costs. Plus there's the obvious legal stuff that would make this a no-go for most companies with anything to lose. The decentralized thing they're doing probably makes way more sense.

How serious compute/storage cost? $10 000 per month? $100 000 per month?
Post reply on HN