Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.3B Worldcat scrape and data science mini-competition

annas-blog.org

Post reply on HN
Print view View original

71–80 of 95 posts

Re: 1.3B Worldcat scrape and data science mini-competition

#71

Post by mannyv »

hedora wrote: post #16

I looked into using Dewey Decimal for a hobby project. OCLC has a de facto monopoly on it due to the Worldcat database. They're a non-profit, but they're supported by having libraries pay a subscription fee for Worldcat. Back when OCLC was founded, the idea that people would want to have a copy of a card catalog for personal use was laughable, so I'm sympathetic to the people that set up their funding model. It's far…

One strange thing about the Dewey Decimal System is that it's copyrighted and libraries pay a fee to use it.

It came out when the Library Hotel in NYC used some of their notation as their room numbers and were sued. Everyone was WTF?

Re: 1.3B Worldcat scrape and data science mini-competition

#72

Post by mannyv »

wordpad25 wrote: post #52
mannyv wrote: post #30

Earlier quoted context omitted.

Yes, but why weren't they first?

It's like asking why wasn't X invented earlier. Google and everyone else had no idea how successful LLMs could be until OpenAI did it.

Untrue. The whole transformer idea came from the goog team

Re: 1.3B Worldcat scrape and data science mini-competition

#73

Post by Tomte »

wiml wrote: post #29
freewizard wrote: post #28

ISBN is the default ID when it comes to book related projects, yes it is convenient but not without its caveats. The often overlooked fact is ISBN was introduced in late 1960s, so books published prior to that obviously does not have that number; and not all countries adopted ISBN from day one, some like China was on its own catalog systems until 1980s; and bc ISBN are usually centralized managed by govt or commercia…

A second issue is that ISBNs identify a specific SKU (different formats will have different ISBNs, different printings may even get different ISBNs, etc), but book-related projects typically want some way to identify "the same book" across all these different formats, printings, sometimes even editions and translations and collections. OCLC IDs are identifying a different space than ISBNs are.

The biggest problem of all is that there are many ISBNs that have been reused for either a later edition of "the same" book or for a totally different book, which should never happen.

Sometimes it is because people are sloppy, sometimes people try to save a little money (because ISBNs cost money).

Re: 1.3B Worldcat scrape and data science mini-competition

#74

Post by dredmorbius »

Question for anyone from Anna's Archive or elsewhere: are catalogue metadata available from national library collections such as the British Library or US Library of Congress?

(I've ... worked a bit with LoC classification and subject headings data, of which publicly-available data are only available in PDF or wordprocessing (MS Word or Wordperfect, if memory serves) formats. Which is ... somewhat unfortunate.)

Re: 1.3B Worldcat scrape and data science mini-competition

#75

Post by dredmorbius »

chatmasta wrote: post #47
RhodesianHunter wrote: post #46

Earlier quoted context omitted.

For real. Openly bragging about exploiting security flaws to scrape out data en-masse, which undoubtedly put massive strain on back-end systems, is a far cry from what is considered legal (politely scraping public information).

I think this is the least of their concerns considering the rest of their activities. I guess they've got a sort of pirate's privilege in that they can openly brag about this stuff since they're already starting from the point of openly flaunting the law. Also, I wouldn't be surprised if there simply are no lawyers working at, for or with Anna's Archive.

Scraping and giving away the content is a better look than scraping and selling it as well.

This is being done for public benefit, not private profit.

Re: 1.3B Worldcat scrape and data science mini-competition

#76

Post by throwawaykasc »

dredmorbius wrote: post #74

Question for anyone from Anna's Archive or elsewhere: are catalogue metadata available from national library collections such as the British Library or US Library of Congress? (I've ... worked a bit with LoC classification and subject headings data, of which publicly-available data are only available in PDF or wordprocessing (MS Word or Wordperfect, if memory serves) formats. Which is ... somewhat unfortunate.)

Check out z39.50, you can use it to search and pull data from most national and university library catalogs. But also be prepared for some code archaeology along the way, as the protocol is around 50 years old.

https://en.wikipedia.org/wiki/Z39.50 https://z-brary.com/ https://github.com/asl2/PyZ3950/blob/master/PyZ3950/zoom.py

Re: 1.3B Worldcat scrape and data science mini-competition

#77

Post by layer8 »

amelius wrote: post #34
bambax wrote: post #18

Earlier quoted context omitted.

It's exactly like this. Publishers get a range, and do whatever they want with it. Also, some agencies sell the numbers range to publishers (I believe the US is in that case) and others give them away at no cost (France). As a result, some small publishers get ISBNs from more liberal agencies than their own country: one can never be 100% sure a French ISBN matches a French publisher for example. It's also possible so…

Why not use UUIDs? People never enter ISBNs manually, anyways. So it might as well be longer string. Or a QR code.

ISBNs contain a check digit, to prevent typos. So you’d have to invent a new format.

Re: 1.3B Worldcat scrape and data science mini-competition

#78

Post by gmcharlt »

greenie_beans wrote: post #65
gmcharlt wrote: post #8

Earlier quoted context omitted.

It's not the software per se, which is generally fit for purpose but not amazing, but the traditions and economics underpinning how libraries maintain their bibliographic metadata. Libraries sharing metadata for their catalogs has a long history, dating back to at least 1902 when the Library of Congress started selling catalog cards for use by other libraries. In the 1960s, the Library of Congress embarked on various…

> (Fortunately for them, there are various alternative ways of getting MARC records for free or very cheap, but nobody has a database more comprehensive than Worldcat.) what are some of these sources? isbndb and open library? proquest?

Many libraries [1], including the likes of the Library of Congress and the National Library of Australia, make their catalogs' MARC records freely available via a library-specific protocol called Z39.50. The Library of Congress makes their catalog metadata available in other ways [2][3] and the Internet Archive has a collection of MARC records as well [4]. There are also a couple commercial services that provide them, and publishers, particularly of digital collections used by libraries, will sometimes supply MARC records (though they tend to be low-quality).

[1] https://irspy.indexdata.com/ [2] https://www.loc.gov/cds/products/marcDist.php [3] https://id.loc.gov/ [4] https://archive.org/details/ol_data

Re: 1.3B Worldcat scrape and data science mini-competition

#79

Post by wolverine876 »

An earlier study that addressed the scope of all published works:

J-B Michel, et al. "Quantitative Analysis of Culture Using Millions of Digitized Books". Science (the journal) (16 Dec 2010). https://www.science.org/doi/10.1126/science.1199644

Their focus was on words but along the way they analyzed the number of published texts, a study that "includes serials and sets but excludes kits, mixed media, and periodicals such as newspapers".

They concluded that the world had published 129 million "editions" (one book may have multiple editions).

Re: 1.3B Worldcat scrape and data science mini-competition

#80

Post by greenie_beans »

this is an obvious prediction, but with the writers class action lawsuit against openai for using their books, the internet will become more closed. it's gonna be so hard to scrape websites in the future. we were trending in this direction before gpt, but gpt exacerbated this and put the issue into the mainstream.
Post reply on HN