Earlier quoted context omitted.
If it's on the internet, and people can access it, then it's public. I would have no expectations for what people do with public data; that just seems like setting yourself up for disappointment.
Is a pirated movie, found on bittorrent, public? IMO, your definition is overbroad
Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
211–220 of 275 posts
Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#212Earlier quoted context omitted.
last.fm used a primitive machine learning algorithm too, else what are you going to rank by?
Did they? I recall similar site back from as far as from 2008. Might be them or something similar. Anyway. I can totally see such site running purely on statistics. Every song, every artists, every genre is a bucket. You listen to a song you put a drop in these buckets. Once there's enough water running we can compare you to other users and their buckets. It might be hard to run it on scale in real time, but c'mon, i…
Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#213Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#214Thank you! Because of this, "The Making of Prince of Persia: Journals 1985–1993" by Jordan Mechner is on its way to my house.
Here’s a human recommendation if you like that you may like these that I’ve read: 1.Sid Meier’s Memoir!: A Life in Computer Games — Sid Meier 2.Source Code: My Beginnings — Bill Gates 3.Build: An Unorthodox Guide to Making Things Worth Making — Tony Fadell 4.Prince of Persia: The Journals — Jordan Mechner 5.A Theory of Fun for Game Design — Raph Koster 6.Ask Iwata: Words of Wisdom from Satoru Iwata, Nintendo’s Legend…
Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#215Earlier quoted context omitted.
I’m not taking sides in this debate, however since feeding whole books into LLMs is considered legal fair use now, I guess these reviews don’t require a permission as well. Would be great to hear a professional lawyer take on this.
The hidden gotcha in the Anthropic judgement (which I think is what you’re referencing?) is that feeding whole books into LLMs is considered legal fair use if you obtain them legitimately . I suspect we need to wait for the NYT (and others) case to be decided before we know whether scraping sites in contravention of their terms is also fair use for LLM training. My own opinion (as someone who creates written content…
Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#216The robots.txt is pretty explicit that this scraping is "disallowed" https://www.goodreads.com/robots.txt So legalities aside, this seems unethical.
I agree. As a frequent reviewer on Goodreads, this feels really icky.
At the same time, everything you ever posted online has already been scraped by hundreds (maybe thousands) of entities and distributed/sold to countless other entities. The only difference is that OP shared his project here.
Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#217Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…
Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#218Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#219Earlier quoted context omitted.
So if you are legally allowed to "adapt, edit or create derivative works from any materials", what's the point of the TOS?
I believe TOS is binding as long as it doesn't conflict with the law. If something is deemed fair use under the law, TOS cannot override those legal rights.
Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
#220Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…
Fairly meaningless in this day and age. Also IIRC scraping legality depends heavily on jurisdiction. Some places take a more permissive view of accessing publicly available information, even if a site's TOS forbids bots. In the US there’s a major precedent [0] which held that scraping public-facing pages isn’t a CFAA "unauthorized access" issue. That’s a big part of why we’ve seen entire venture-backed scraping compa…
This was part of the terms of the settlement.