Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.3B Worldcat scrape and data science mini-competition

annas-blog.org

Post reply on HN
Print view View original

31–40 of 95 posts

Re: 1.3B Worldcat scrape and data science mini-competition

#31

Post by harveywi »

>Over the past year, we’ve meticulously scraped all Worldcat records. At first, we hit a lucky break. Worldcat was just rolling out their complete website redesign (in Aug 2022). This included a substantial overhaul of their backend systems, introducing many security flaws. We immediately seized the opportunity, and were able scrape hundreds of millions (!) of records in mere days.

>After that, security flaws were slowly fixed one by one, until the final one we found was patched about a month ago. By that time we had pretty much all records, and were only going for slightly higher quality records.

OCLC carelessly fiddlefarted around with their moat and lost it. Poof!

Re: 1.3B Worldcat scrape and data science mini-competition

#32

Post by skuxxlife »

DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

For US/UK/NZ/Aus/SA, ISBNs are granted through Bowker who does maintain their "Books In Print" data set that, in theory, contains metadata for all of the ISBNs they've granted. In practice though it's a mess. It's expensive to access and relies on publishers to enter in accurate and consistent metadata, which is...variable in quality to say the least. Often publishers buy blocks of ISBNs to use later so no metadata is entered up front and has to be pushed back to Bowker at a later date. To be somewhat fair to Bowker, the history of ISBNs far predates modern data standards and I can imagine wrangling publishers to get accurate data is a difficult task. But on the other hand, you'd think they'd have a lot to gain for doing it right. As someone who runs a book website, it is endlessly frustrating.

Re: 1.3B Worldcat scrape and data science mini-competition

#34

Post by amelius »

bambax wrote: post #18
sevenseventen wrote: post #12

Earlier quoted context omitted.

There are regional ISBN agencies. The US agency, Bowker, assigns ISBN prefixes by publisher, and publishers assign within their prefix as they please. They're supposed to use one ISBN per edition and format, but many publishers use ISBN as a kind of SKU so you can't 100% count on that. If that sounds sloppy...I went to publishing conferences fairly regularly from the late 90's into the teens, and I never saw a progra…

It's exactly like this. Publishers get a range, and do whatever they want with it. Also, some agencies sell the numbers range to publishers (I believe the US is in that case) and others give them away at no cost (France). As a result, some small publishers get ISBNs from more liberal agencies than their own country: one can never be 100% sure a French ISBN matches a French publisher for example. It's also possible so…

Why not use UUIDs?

People never enter ISBNs manually, anyways. So it might as well be longer string. Or a QR code.

Re: 1.3B Worldcat scrape and data science mini-competition

#35

Post by dwringer »

amelius wrote: post #34
bambax wrote: post #18

Earlier quoted context omitted.

It's exactly like this. Publishers get a range, and do whatever they want with it. Also, some agencies sell the numbers range to publishers (I believe the US is in that case) and others give them away at no cost (France). As a result, some small publishers get ISBNs from more liberal agencies than their own country: one can never be 100% sure a French ISBN matches a French publisher for example. It's also possible so…

Why not use UUIDs? People never enter ISBNs manually, anyways. So it might as well be longer string. Or a QR code.

As someone who's worked in the field of used books, I can say from personal experience that's not quite true. It's pretty common to have to type in an ISBN for a variety of reasons. Many times the barcodes have been covered up or defaced, and many publications don't have barcodes in the first place.

Re: 1.3B Worldcat scrape and data science mini-competition

#36

Post by amelius »

dwringer wrote: post #35
amelius wrote: post #34

Earlier quoted context omitted.

Why not use UUIDs? People never enter ISBNs manually, anyways. So it might as well be longer string. Or a QR code.

As someone who's worked in the field of used books, I can say from personal experience that's not quite true. It's pretty common to have to type in an ISBN for a variety of reasons. Many times the barcodes have been covered up or defaced, and many publications don't have barcodes in the first place.

It was true, for sure. But the question is, is it still true?

Re: 1.3B Worldcat scrape and data science mini-competition

#37

Post by mgr86 »

hedora wrote: post #16

I looked into using Dewey Decimal for a hobby project. OCLC has a de facto monopoly on it due to the Worldcat database. They're a non-profit, but they're supported by having libraries pay a subscription fee for Worldcat. Back when OCLC was founded, the idea that people would want to have a copy of a card catalog for personal use was laughable, so I'm sympathetic to the people that set up their funding model. It's far…

I have been told that my organization developed a system, HABS, that pre-dated OCLC [0]. That OCLC used this system as an inspiration. However, I cannot confirm this. Closest I can do is to find a footnote that Thanks Fred Kilgore, the founder of OCLC [1]. I should reach out to Koh, a friend of a friend, while she is still alive to confirm the story. Nevertheless, we have a collection of punch cards in a dusty room in an attic that was once the HABS system. I think it is a pretty fascinating legacy, and I wish it was better preserved.

[0] https://journals.sagepub.com/doi/pdf/10.1177/106939716900400... [1]https://journals.sagepub.com/doi/abs/10.1177/106939717300800...

Re: 1.3B Worldcat scrape and data science mini-competition

#38

Post by raybb »

wiml wrote: post #29
freewizard wrote: post #28

ISBN is the default ID when it comes to book related projects, yes it is convenient but not without its caveats. The often overlooked fact is ISBN was introduced in late 1960s, so books published prior to that obviously does not have that number; and not all countries adopted ISBN from day one, some like China was on its own catalog systems until 1980s; and bc ISBN are usually centralized managed by govt or commercia…

A second issue is that ISBNs identify a specific SKU (different formats will have different ISBNs, different printings may even get different ISBNs, etc), but book-related projects typically want some way to identify "the same book" across all these different formats, printings, sometimes even editions and translations and collections. OCLC IDs are identifying a different space than ISBNs are.

What you're referring to is sometimes referred to as a "work" vs "edition"

https://openlibrary.org/help/faq/editing#work-edition

Re: 1.3B Worldcat scrape and data science mini-competition

#39

Post by neilv »

How does Anna's Archive keep their all their lawyers from quitting?

> Even though OCLC is a non-profit, their business model requires protecting their database. Well, we’re sorry to say, friends at OCLC, we’re giving it all away. :-) [...]

> This included a substantial overhaul of their backend systems, introducing many security flaws. We immediately seized the opportunity, and were able scrape hundreds of millions (!) of records in mere days. [...]

> PS: We do want to give a genuine shout-out to the Worldcat team. Even though it was a small tragedy that your data was locked up, you did an amazing job at getting 30,000 libraries on board to share their metadata with you. As with many of our releases, we could not have done it without the decades of hard work you put into building the collections that we now liberate. Truly: thank you.

Re: 1.3B Worldcat scrape and data science mini-competition

#40

Post by bambax »

amelius wrote: post #36
dwringer wrote: post #35

Earlier quoted context omitted.

As someone who's worked in the field of used books, I can say from personal experience that's not quite true. It's pretty common to have to type in an ISBN for a variety of reasons. Many times the barcodes have been covered up or defaced, and many publications don't have barcodes in the first place.

It was true, for sure. But the question is, is it still true?

Schools often ask for a specific edition of a classic book and the only way to be reasonably sure you're buying the correct one is to search by ISBN.
Post reply on HN