Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.3B Worldcat scrape and data science mini-competition

annas-blog.org

Post reply on HN
Print view View original

11–20 of 95 posts

Re: 1.3B Worldcat scrape and data science mini-competition

#11

Post by yorwba »

DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

Companies can get a range of ISBNs before deciding what to publish under each ISBN, or whether to publish something at all. So the authorities assigning ISBNs don't necessarily know what they're being used for.

Re: 1.3B Worldcat scrape and data science mini-competition

#12

Post by sevenseventen »

DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

There are regional ISBN agencies. The US agency, Bowker, assigns ISBN prefixes by publisher, and publishers assign within their prefix as they please. They're supposed to use one ISBN per edition and format, but many publishers use ISBN as a kind of SKU so you can't 100% count on that.

If that sounds sloppy...I went to publishing conferences fairly regularly from the late 90's into the teens, and I never saw a program that didn't have at least one session or panel titled something like "Publishers must improve their metadata practices."

Re: 1.3B Worldcat scrape and data science mini-competition

#13

Post by gmcharlt »

DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

ISBNs are messy.

The International ISBN Agency coordinates assigning ISBN ranges to national agencies, who in turn will assign subranges to publishers. The publishers in turn assign specific numbers to their own works. However, the international agency does not itself maintain a universal database of assigned ISBNs - the most it operates is a global database of publishers and their assigned ranges. And since it's the publishers who are assigning numbers from their allocations, various errors can crop up, including reusing ISBNs for different works and failing to issue distinct ISBNs for different formats. (For example, if you publish hardcover, paperbook, and ebook versions of a book, you should assign three ISBNs. That rule is not always observed.)

Also, libraries hold many books that long predate ISBNs; it wasn't until 1965 that the immediate predecessor of the ISBN, the SBN, was a twinkle in a bookseller's eye.

Re: 1.3B Worldcat scrape and data science mini-competition

#14

Post by crtasm »

DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

There's lots of agencies, they hand out blocks of numbers from their allocation. Seems there's no central database for the metadata:

https://www.isbn-international.org/content/isbn-users-manual... ISBN_FAQs_to_7ed_Manual_Absolutely_final.docx

> Will people in other countries be able to search for my books in search engines in those countries?

> This does not happen automatically ... In order for your book to be listed in other countries you should contact the respective ISBN Agency and ask them for details of how to be entered into their national catalogue for books in circulation (books in print). Sometimes you will have to obtain a distributor from that country or have an address in that country before this is possible. In some circumstances in order to be listed, the book must be in the language of that country. As well as catalogues of books in circulation, you may also want to ensure that you are listed by internet retailers... . Again, you will need to contact each of these organisations directly (including each separate international branch) with details of your book.

Re: 1.3B Worldcat scrape and data science mini-competition

#15

Post by gmcharlt »

kinos wrote: post #3
raybb wrote: post #2

From the end: > We do want to give a genuine shout-out to the Worldcat team. Even though it was a small tragedy that your data was locked up, you did an amazing job at getting 30,000 libraries on board to share their metadata with you. I wonder what the story is behind Worldcat getting so many libraries across the world on board? I don't know much about the software but it must be pretty compelling.

they probably have a good interface for personal library tracking

It's not a LibraryThing or GoodReads; it's meant for libraries that are institutions. That said, I don't think there is anything stopping an individual person signing up and entering their collection, but there would be no point in paying the fees unless you had (say) a unique scholarly collection and wanted to lend books to other libraries - and if so, in the long run you'd likely be better off seeing if a library wanted to acquire your collection.

Re: 1.3B Worldcat scrape and data science mini-competition

#16

Post by hedora »

I looked into using Dewey Decimal for a hobby project. OCLC has a de facto monopoly on it due to the Worldcat database. They're a non-profit, but they're supported by having libraries pay a subscription fee for Worldcat.

Back when OCLC was founded, the idea that people would want to have a copy of a card catalog for personal use was laughable, so I'm sympathetic to the people that set up their funding model. It's far cheaper for a library to subscribe to Worldcat than to hire a team to maintain such a database, so it created a win-win situation.

However, keeping the world's books' metadata a secret (and leaving control in the hands of a monopoly) is an anachronism.

It's well past the time when someone (such as an international coalition of Libraries of Congress) should figure out how to sustainably fund OCLC while also releasing their work into the public domain.

Re: 1.3B Worldcat scrape and data science mini-competition

#17

Post by jszymborski »

sevenseventen wrote: post #12
DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

There are regional ISBN agencies. The US agency, Bowker, assigns ISBN prefixes by publisher, and publishers assign within their prefix as they please. They're supposed to use one ISBN per edition and format, but many publishers use ISBN as a kind of SKU so you can't 100% count on that. If that sounds sloppy...I went to publishing conferences fairly regularly from the late 90's into the teens, and I never saw a progra…

Sounds a bit like how DOIs are assigned.

Re: 1.3B Worldcat scrape and data science mini-competition

#18

Post by bambax »

sevenseventen wrote: post #12
DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

There are regional ISBN agencies. The US agency, Bowker, assigns ISBN prefixes by publisher, and publishers assign within their prefix as they please. They're supposed to use one ISBN per edition and format, but many publishers use ISBN as a kind of SKU so you can't 100% count on that. If that sounds sloppy...I went to publishing conferences fairly regularly from the late 90's into the teens, and I never saw a progra…

It's exactly like this. Publishers get a range, and do whatever they want with it.

Also, some agencies sell the numbers range to publishers (I believe the US is in that case) and others give them away at no cost (France). As a result, some small publishers get ISBNs from more liberal agencies than their own country: one can never be 100% sure a French ISBN matches a French publisher for example.

It's also possible some publishers re-use old numbers or affix the same number to different releases/editions of a book.

It's a mess.

But a global centralized system would probably be way worse, so we have to live with that mess.

Re: 1.3B Worldcat scrape and data science mini-competition

#19

Post by bambax »

gmcharlt wrote: post #13
DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

ISBNs are messy. The International ISBN Agency coordinates assigning ISBN ranges to national agencies, who in turn will assign subranges to publishers. The publishers in turn assign specific numbers to their own works. However, the international agency does not itself maintain a universal database of assigned ISBNs - the most it operates is a global database of publishers and their assigned ranges. And since it's the…

Yes.

And while in most countries you can't properly publish a book without an ISBN (ie, have it sold in bookshops), you can publish a Kindle book without it (if you opt to only offer the ebook).

That leaves a huge part of publications completely out of the system. Kindle-only books are on Amazon servers and nowhere else.

Post reply on HN