Live data from Hacker News

Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

book.sv

211–220 of 275 posts

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#211
post #192

Earlier quoted context omitted.

If it's on the internet, and people can access it, then it's public. I would have no expectations for what people do with public data; that just seems like setting yourself up for disappointment.

Is a pirated movie, found on bittorrent, public? IMO, your definition is overbroad

If it's on bittorrent then, yes, it's public. It doesn't matter if you intended it to be or not, it's publicly accessible, therefore it's public.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#212
post #107
post #103

Earlier quoted context omitted.

last.fm used a primitive machine learning algorithm too, else what are you going to rank by?

Did they? I recall similar site back from as far as from 2008. Might be them or something similar. Anyway. I can totally see such site running purely on statistics. Every song, every artists, every genre is a bucket. You listen to a song you put a drop in these buckets. Once there's enough water running we can compare you to other users and their buckets. It might be hard to run it on scale in real time, but c'mon, i…

[dead]

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#214
post #148

Thank you! Because of this, "The Making of Prince of Persia: Journals 1985–1993" by Jordan Mechner is on its way to my house.

Here’s a human recommendation if you like that you may like these that I’ve read: 1.Sid Meier’s Memoir!: A Life in Computer Games — Sid Meier 2.Source Code: My Beginnings — Bill Gates 3.Build: An Unorthodox Guide to Making Things Worth Making — Tony Fadell 4.Prince of Persia: The Journals — Jordan Mechner 5.A Theory of Fun for Game Design — Raph Koster 6.Ask Iwata: Words of Wisdom from Satoru Iwata, Nintendo’s Legend…

Thank you for the recs, Masters of Doom really lit a fire inside of me as a young programmer. I'd also recommend "Soul of a New Machine" by Tracy Kidder.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#215

Earlier quoted context omitted.

I’m not taking sides in this debate, however since feeding whole books into LLMs is considered legal fair use now, I guess these reviews don’t require a permission as well. Would be great to hear a professional lawyer take on this.

The hidden gotcha in the Anthropic judgement (which I think is what you’re referencing?) is that feeding whole books into LLMs is considered legal fair use if you obtain them legitimately . I suspect we need to wait for the NYT (and others) case to be decided before we know whether scraping sites in contravention of their terms is also fair use for LLM training. My own opinion (as someone who creates written content…

Goodreads offers those reviews up publicly by serving them from their webservers to anyone who asks for it.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#216

The robots.txt is pretty explicit that this scraping is "disallowed" https://www.goodreads.com/robots.txt So legalities aside, this seems unethical.

I agree. As a frequent reviewer on Goodreads, this feels really icky.

You are right.

At the same time, everything you ever posted online has already been scraped by hundreds (maybe thousands) of entities and distributed/sold to countless other entities. The only difference is that OP shared his project here.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#217

Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…

At what point are they feeding reviews into an LLM? From what I got the only personal data they're using is which user read which books.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#218
For the intersect page you probably want to order the users by the size of their shelf. For some more obscure combinations I'm mostly getting users who read 10,000s of books, which is less useful than the users with <1000 books.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#219

Earlier quoted context omitted.

So if you are legally allowed to "adapt, edit or create derivative works from any materials", what's the point of the TOS?

I believe TOS is binding as long as it doesn't conflict with the law. If something is deemed fair use under the law, TOS cannot override those legal rights.

Legal rights are signed away all the time in contracts though.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#220
post #135

Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…

Fairly meaningless in this day and age. Also IIRC scraping legality depends heavily on jurisdiction. Some places take a more permissive view of accessing publicly available information, even if a site's TOS forbids bots. In the US there’s a major precedent [0] which held that scraping public-facing pages isn’t a CFAA "unauthorized access" issue. That’s a big part of why we’ve seen entire venture-backed scraping compa…

> CFAA liability: hiQ stipulates that LinkedIn experienced losses sufficient to, and “may establish liability” under a CFAA civil claim “based on hiQ’s data collection practices and based on hiQ’s direct access to password-protected pages on LinkedIn’s platforms using fake accounts.”

This was part of the terms of the settlement.

Post reply on HN