Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.3B Worldcat scrape and data science mini-competition

annas-blog.org

Post reply on HN
Print view View original

61–70 of 95 posts

Re: 1.3B Worldcat scrape and data science mini-competition

#61

Post by neilv »

chatmasta wrote: post #47
RhodesianHunter wrote: post #46

Earlier quoted context omitted.

For real. Openly bragging about exploiting security flaws to scrape out data en-masse, which undoubtedly put massive strain on back-end systems, is a far cry from what is considered legal (politely scraping public information).

I think this is the least of their concerns considering the rest of their activities. I guess they've got a sort of pirate's privilege in that they can openly brag about this stuff since they're already starting from the point of openly flaunting the law. Also, I wouldn't be surprised if there simply are no lawyers working at, for or with Anna's Archive.

They could possibly get important ongoing support from institutions like the EFF and Berkman.

But I think that's less likely if they're gleefully bragging about civil and criminal liabilities.

And also chilling future data-sharing willingness for other orgs (because someone is coming along and ignoring assurances).

Re: 1.3B Worldcat scrape and data science mini-competition

#62

Post by partytax »

hedora wrote: post #16

I looked into using Dewey Decimal for a hobby project. OCLC has a de facto monopoly on it due to the Worldcat database. They're a non-profit, but they're supported by having libraries pay a subscription fee for Worldcat. Back when OCLC was founded, the idea that people would want to have a copy of a card catalog for personal use was laughable, so I'm sympathetic to the people that set up their funding model. It's far…

The Melville Decimal System popularized by https://www.librarything.com may be of interest.

Here's an explanation from their footer: "Although Dewey invented his system in 1876, recent editions of his system are in copyright. LibraryThing's Melvil Decimal System is based on the classification work of libraries around the world, whose assignments are not copyrightable. The "schedules" (the words that describe the numbers) come from a pre-copyright edition of his system, John Mark Ockerbloom's Free Decimal System, and member contributions."

Re: 1.3B Worldcat scrape and data science mini-competition

#63

Post by dredmorbius »

empthought wrote: post #50
IKantRead wrote: post #45

Earlier quoted context omitted.

OCLC is a parasitic company masquerading as a "membership cooperative". Libraries (often publicly funded) produce all the work, OCLC claims ownership of the results of that work, Libraries pay to get it back (but they do get a discount if they contribute). The only reason OCLC continues to exist is because libraries don't have the support or resources to fight them. It's very similar to the Elsevier issue in academic…

I mean you’ve clearly read Aaron Swartz’s diatribes, but you also clearly have no clue about OCLC’s business model. The catalog data is intellectually interesting, but the value is in the holdings data and more importantly the interlibrary loan service it enables. OCLC is exactly what happens when the libraries want to avoid another EBSCOhost or Proquest situation with ILL.

What were the EBSOhost / ProQuest situations, if you don't mind?

Re: 1.3B Worldcat scrape and data science mini-competition

#64

Post by dredmorbius »

freewizard wrote: post #28

ISBN is the default ID when it comes to book related projects, yes it is convenient but not without its caveats. The often overlooked fact is ISBN was introduced in late 1960s, so books published prior to that obviously does not have that number; and not all countries adopted ISBN from day one, some like China was on its own catalog systems until 1980s; and bc ISBN are usually centralized managed by govt or commercia…

Along with other limiations, ISBN are, well, book numbers. They're specific to books, and exclude many other forms of published materials.

OCLC spans books, articles, audio recordings, videos, and other catalogued artefacts and documents.

Re: 1.3B Worldcat scrape and data science mini-competition

#65

Post by greenie_beans »

gmcharlt wrote: post #8
raybb wrote: post #2

From the end: > We do want to give a genuine shout-out to the Worldcat team. Even though it was a small tragedy that your data was locked up, you did an amazing job at getting 30,000 libraries on board to share their metadata with you. I wonder what the story is behind Worldcat getting so many libraries across the world on board? I don't know much about the software but it must be pretty compelling.

It's not the software per se, which is generally fit for purpose but not amazing, but the traditions and economics underpinning how libraries maintain their bibliographic metadata. Libraries sharing metadata for their catalogs has a long history, dating back to at least 1902 when the Library of Congress started selling catalog cards for use by other libraries. In the 1960s, the Library of Congress embarked on various…

> (Fortunately for them, there are various alternative ways of getting MARC records for free or very cheap, but nobody has a database more comprehensive than Worldcat.)

what are some of these sources? isbndb and open library? proquest?

Re: 1.3B Worldcat scrape and data science mini-competition

#66

Post by versteegen »

3abiton wrote: post #54
sterlind wrote: post #49

It's infuriating seeing non-profits gatekeep datasets that were compiled with grant money. At least Elsevier doesn't present itself as a charity. I was recently trying to get my hands on the Switchboard and Fisher conversational speech datasets. Both were funded by DARPA grants, and maintained by the non-profit LDC, which charges you thousands of dollars for access (and no discounts for individual researchers) - that…

I hope one day there will a piratebay for datasets (the pile) and ai models (llama)

Good news, such a site was launched very recently: https://www.thenose.cc/

There's plenty of discussion about it by 'nostril' here on HN: https://news.ycombinator.com/threads?id=nostril

Re: 1.3B Worldcat scrape and data science mini-competition

#67

Post by greenie_beans »

dwringer wrote: post #35
amelius wrote: post #34

Earlier quoted context omitted.

Why not use UUIDs? People never enter ISBNs manually, anyways. So it might as well be longer string. Or a QR code.

As someone who's worked in the field of used books, I can say from personal experience that's not quite true. It's pretty common to have to type in an ISBN for a variety of reasons. Many times the barcodes have been covered up or defaced, and many publications don't have barcodes in the first place.

same

Re: 1.3B Worldcat scrape and data science mini-competition

#68

Post by [deleted] »

amelius wrote: post #36
dwringer wrote: post #35

Earlier quoted context omitted.

As someone who's worked in the field of used books, I can say from personal experience that's not quite true. It's pretty common to have to type in an ISBN for a variety of reasons. Many times the barcodes have been covered up or defaced, and many publications don't have barcodes in the first place.

It was true, for sure. But the question is, is it still true?

[deleted]

Re: 1.3B Worldcat scrape and data science mini-competition

#69

Post by greenie_beans »

skuxxlife wrote: post #32
DoctorOetker wrote: post #5

> We scraped ISBNdb, and downloaded the Open Library dataset, but the results were unsatisfactory. The main problem was that there was not a ton of overlap of ISBNs. What prevents ISBN number collisions between authors? Is there a central authority assigning them, or is there say a national prefix, with each government assigning ISBN's for local publications (perhaps delegating this to another body in that nation). S…

For US/UK/NZ/Aus/SA, ISBNs are granted through Bowker who does maintain their "Books In Print" data set that, in theory, contains metadata for all of the ISBNs they've granted. In practice though it's a mess. It's expensive to access and relies on publishers to enter in accurate and consistent metadata, which is...variable in quality to say the least. Often publishers buy blocks of ISBNs to use later so no metadata i…

what is your website?

Re: 1.3B Worldcat scrape and data science mini-competition

#70

Post by [deleted] »

sterlind wrote: post #49

It's infuriating seeing non-profits gatekeep datasets that were compiled with grant money. At least Elsevier doesn't present itself as a charity. I was recently trying to get my hands on the Switchboard and Fisher conversational speech datasets. Both were funded by DARPA grants, and maintained by the non-profit LDC, which charges you thousands of dollars for access (and no discounts for individual researchers) - that…

[deleted]
Post reply on HN