Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

#31

Post by bbor »

culi wrote: post #25

I find WikiData to be perfect for aggregating identifiers. I mostly work with species names and it's perfect for getting the iNaturalist, GBIF, Open Tree of Life, Catalogue of Life, etc identities all in one query I haven't tried it for books. I imagine it's not sufficiently complete to serve as a backbone but a quick look at an example book gives me the ids for OpenLibrary, Librarything, Goodreads, Bing, and even ni…

Cracks me up that OP is trying Anna's Archive before Wikidata, NGL! Both great sources, though.

I recently (a year ago... wow) dipped my toe into the world of library science through Wikidata, and was shocked at just how complex it is. OP's work looks really solid, but I hope they're aware of how mature the field is!

For illustration, here are just the book-relevant ID sources I focused on from Wikidata:

  ARCHIVERS: 
  Library of Congress Control Number    `P1144` (173M)
  Open Library                          `P648`  (39M)
  Online Computer Library Center        `P10832` (10M)
  German National Library               `P227`  (44M)
  Smithsonian Institute                 `P7851` (155M)
  Smitsonian Digital Ark                `P9473` (3M)
  U.S. Office of Sci. & Tech. Info.     `P3894`

  PUBLISHERS:
  Google Books                          `P675`  (1M)
  Project Gutenberg                     `P2034` (70K)
  Amazon                                `P5749`

  CATALOGUERS:
  International Standard Book Number    `P212`
  Wikidata                              `P8379` (115B)
  EU Knowledge Graph                    `P11012`
  Factgrid Database                     `P10787` (0.4M)
  Google Knowledge Graph                `P2671` (500B)

Re: Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

#32

Post by jamesponddotco »

eichin wrote: post #27

Tried throwing a batch of known-to-be-in-Amazon ISBN's through (from a recent "export my data", so even if they're old amazon fundamentally knows them.) Got 500's for a handful of the first hundred, then a bunch of 502/503s (so, single threaded, but part of the HN hug to death, sorry!) (Only the first 4 or so were json errors, the rest were html-from-nginx, if that matters.)

No hug of death, the server is sitting at 3% CPU usage under current load; it seems someone found a bug that triggered a panic, and systemd failed to restart the service because the PID file wasn't removed. Fixed now, should be back online :)

Re: Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

#34

Post by jamesponddotco »

ocdtrekkie wrote: post #23

Library of Congress data seems like a huge omission especially for something named after a librarian. ;) It is a very easy API to consume too.

I didn't look into it yet because I assumed the current extractors had the information from them, but it's in my list of future extractors!

Re: Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

#35

Post by jamesponddotco »

bbor wrote: post #31
culi wrote: post #25

I find WikiData to be perfect for aggregating identifiers. I mostly work with species names and it's perfect for getting the iNaturalist, GBIF, Open Tree of Life, Catalogue of Life, etc identities all in one query I haven't tried it for books. I imagine it's not sufficiently complete to serve as a backbone but a quick look at an example book gives me the ids for OpenLibrary, Librarything, Goodreads, Bing, and even ni…

Cracks me up that OP is trying Anna's Archive before Wikidata, NGL! Both great sources, though. I recently (a year ago... wow) dipped my toe into the world of library science through Wikidata, and was shocked at just how complex it is. OP's work looks really solid, but I hope they're aware of how mature the field is! For illustration, here are just the book-relevant ID sources I focused on from Wikidata: ARCHIVERS: L…

Not gonna lie, I didn't even know Wikidata existed until now. I'll look into it today and create a ticket for a new extractor.

Thanks for letting me know!

Re: Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

#36

Post by jamesponddotco »

karlding wrote: post #22

Do you handle books with no ISBN? I’ve recently acquired some photo books that don’t appear to have any ISBN but are listed on WorldCat and have OCLC Numbers and are catalogued in the Japanese National Diet Library. Not sure if they actually don't have ISBNs or if I just haven't been able to find them, but from what I got from some research it's quite common for self-published books.

No, right now you need an ISBN to search for a book. At a later date I'll implement search by title or author, which should help with this use case.

Re: Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

#37

Post by jamesponddotco »

fc417fc802 wrote: post #20
jamesponddotco wrote: post #16

Earlier quoted context omitted.

I only tested English and Brazilian Portuguese so far, and Brazilian Portuguese worked, with translator information included.

You're most likely to run into issues with non-latin languages. Particularly picograms and the associated schemes for how to interpret them in a context sensitive manner. Substring search for example is likely to be broken in my experience.

Can you provide an example?

Re: Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

#38

Post by mehdi1964 »

Nice approach! Merging metadata from multiple sources is tricky, especially handling conflicts like titles and covers. Curious how you plan to handle scalability as your database grows—caching helps, but will the naive field strategies hold with thousands of books?

Re: Show HN: Librario, a book metadata API that aggregates G Books, ISBNDB, and more

#40

Post by zozbot234 »

culi wrote: post #30
jamesponddotco wrote: post #3

Earlier quoted context omitted.

Hardcover and ISBNDB have a good amount of data, with Hardcover being excellent for getting good covers and genres. I'm hoping Goodreads and Anna's Archive will help fill in the gaps, especially since Anna's Archive have gigantic database dumps available[1]. [1]: https://todo.sr.ht/~pagina394/librario/12

You should also consider OpenLibrary and LibraryThing. Both of which have good coverage on WikiData which also aggregates identifiers. In fact, now that I think about it, you could also contribute your work to WikiData. I don't see ISBNdb ids on WikiData so you could write a script to make those contributions. Then anyone else using WikiData for this sort of thing can benefit from your work

ISBNid seems to use ISBN-13 for its unique identifier, which is just Property:P212 on Wikidata.
Post reply on HN