Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

book.sv

Post reply on HN
Print view View original

221–230 of 275 posts

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#221

Post by hrimfaxi »

SahAssar wrote: post #153
esskay wrote: post #135

Earlier quoted context omitted.

Fairly meaningless in this day and age. Also IIRC scraping legality depends heavily on jurisdiction. Some places take a more permissive view of accessing publicly available information, even if a site's TOS forbids bots. In the US there’s a major precedent [0] which held that scraping public-facing pages isn’t a CFAA "unauthorized access" issue. That’s a big part of why we’ve seen entire venture-backed scraping compa…

From that article: > However, after further appeal in another court, hiQ was found to be in breach of LinkedIn's terms, and there was a settlement. So why would the same not apply here?

They settled out of court, that doesn't mean that they were found to be in breach of the terms.

These were some of the notable elements (worth noting that none mention breaching terms of service):

> Damages: Judgment in the amount of $500,000 is entered against hiQ, with all other monetary relief waived.

> CFAA liability: hiQ stipulates that LinkedIn experienced losses sufficient to, and “may establish liability” under a CFAA civil claim “based on hiQ’s data collection practices and based on hiQ’s direct access to password-protected pages on LinkedIn’s platforms using fake accounts.”

> California “CFAA”: hiQ stipulates that LinkedIn “may establish civil liability” under California’s state-law counterpart to the CFAA based on hiQ’s data collection practices, use of fake accounts and other means to evade detection by LinkedIn, hiQ’s direct access to password-protected pages on LinkedIn’s platforms using fake accounts, and hiQ’s unauthorized commercial use of data.

> Trespass: hiQ stipulates that LinkedIn has established judgment as to liability under California law for the common law torts of trespass to chattels and misappropriation.

> Irreparable harm: hiQ stipulates that LinkedIn has established that it has suffered an irreparable injury and that LinkedIn satisfied the remaining factors and is entitled to a permanent injunction.

https://natlawreview.com/article/hiq-and-linkedin-reach-prop...

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#222

Post by sosuke »

Feature request: Combine book series into a single entity. Bummer getting recommendations for another book in the same series as one I already liked and read.

Feature request: Exclude books already in shelf. This is harder I'm sure. I've got 1146 books in my Read shelf.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#223

Post by contravariant »

diffeomorphism wrote: post #141

The robots.txt is pretty explicit that this scraping is "disallowed" https://www.goodreads.com/robots.txt So legalities aside, this seems unethical.

If it's unethical it's not because of what the robots.txt says.

Blindly violating it is bad manners, but deliberately scraping a single website over a month isn't the worst.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#225

Post by Ntrails »

blehn wrote: post #62

You should filter out authors from the input books in the output. If liked a book by an author, surely I'd read more of their work if I wanted to — recommending them isn't helpful. Along the same lines, I think interesting recommendations tend to be the ones that (1) I like and (2) I didn't expect. The more similar the recommendations are to the input, the more likely I already know them, and the more likely to creat…

Agree entirely - more excluding series than authors but both should be options.

I also i need a way to describe its recommendations as "meh". For example, if I put Gone Girl in, I get Girl on a Train. Which, personally, I thought was bad. I want to exclude that from all future rec sets, and ideally align my preferences to the intersection of liked A and disliked B. vOv

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#227

Post by MichaelBosworth »

caconym_ wrote: post #190
MichaelBosworth wrote: post #165

Earlier quoted context omitted.

What expectation of confidentiality are you ascribing to people having posted publicly accessible opinions on the internet? Out of curiosity, is your point about TOS out of concern for the poster or for Goodreads?

What does that comment have to do with confidentiality?

That he viewed a review on Goodreads as the reviewer’s intellectual property hadn’t occurred to me. I see why, in aggregate, many such opinions become valuable, but the whole is more than the sum of its parts.

So does it feel to you guys like your comments, say, here in this Hacker News thread should be considered effectively copyrighted as your personal IP?

If so, do you feel the same way about opinions you share out in a supermarket or on the street?

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#228

Post by mulakosag »

dbl000 wrote: post #50

Echoing what everyone else has said here - awesome site, love how fast it was. I did notice that when I put in a single book in a series (in my case Going Postal, Discworld #33) that tended to dominate the rest of the selection. That does make sense, but I don't want recommendations for a series I'm already well into. Also noticed that a few books (Spycraft by Nadine Akkerman and Pete Langman, Tribalism is Dumb by An…

Going Postal is awesome. The flood of the mails and the test to be the post master where you would have a slide mail into the hole where a vicious dog is barking.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#229

Post by saaaaaam »

IncreasePosts wrote: post #215
saaaaaam wrote: post #181

Earlier quoted context omitted.

The hidden gotcha in the Anthropic judgement (which I think is what you’re referencing?) is that feeding whole books into LLMs is considered legal fair use if you obtain them legitimately . I suspect we need to wait for the NYT (and others) case to be decided before we know whether scraping sites in contravention of their terms is also fair use for LLM training. My own opinion (as someone who creates written content…

Goodreads offers those reviews up publicly by serving them from their webservers to anyone who asks for it.

Sorry, I don’t understand the point you’re making. I know that these are publicly available - the point I was making, drawing off the parent comment, is that where it has been deemed fair use in copyright to use books to train LLMs when the content has been legitimately obtained then a similar assessment might apply for this sort of ingestion.

If content is publicly available that does not necessarily mean it’s free of copyright control: the justification for using the reviews to train an LLM would be based on the fact that fair use means it is not an infringement of copyright. But if the publisher has terms that forbid scraping then that may mean the fair use argument is undermined if it is precedent in the content being legitimately obtained. I’m not a lawyer but it’s quite easy to see how “books can be used for LLM training under fair use but not if you pirate them” extends to “content on the web can be used for LLM training under fair use but not if you’ve breached the terms set out by the publisher”.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#230

Post by bravoetch »

tonyhart7 wrote: post #170
voidUpdate wrote: post #145

Earlier quoted context omitted.

I'd be impressed if a youtuber could read 3 billion reviews and recommend books to you based on that

what about youtuber that build a machine that scrape 3 billions books and make recommendation based on the data????

Skip that step. This project enables a Youtuber that automates pulling related booklists from this site, and uses AI to make the recommendation videos. Thousands of videos.
Post reply on HN