Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

book.sv

Post reply on HN
Print view View original

151–160 of 275 posts

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#151

Post by costco »

brailsafe wrote: post #115
costco wrote: post #89

Earlier quoted context omitted.

I would agree the results are generally OK but do not feel magical in most cases (I think in some specific cases they do though). The results can be not great if you add books across many disciplines. For instance if you add "The Elements of Typographic Style" and "The Design of Everyday Things" ( https://book.sv/#671857,18518 ), you do get "Grid Systems in Graphic Design" but under its German name "Rastersysteme für…

Wow, quite impressive actually, some of those additional entries are quite good as well, with Ellen Lupton, Steve Krugman, Erik Spiekermann, and then also 1984 and Mark Manson for some reason. Would the latter few be because they're just super widely read/recommended/connected regardless of genre? Great project anyway, thanks for sharing and responding to my stream of consciousness, I certainly didn't mean to be insu…

Yeah, the latter are just because they are popular. If you have 3+ books you tend to get less random popular books included

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#152

Post by voidUpdate »

esskay wrote: post #135
voidUpdate wrote: post #124

Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…

Fairly meaningless in this day and age. Also IIRC scraping legality depends heavily on jurisdiction. Some places take a more permissive view of accessing publicly available information, even if a site's TOS forbids bots. In the US there’s a major precedent [0] which held that scraping public-facing pages isn’t a CFAA "unauthorized access" issue. That’s a big part of why we’ve seen entire venture-backed scraping compa…

So if you are legally allowed to "adapt, edit or create derivative works from any materials", what's the point of the TOS?

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#153

Post by SahAssar »

esskay wrote: post #135
voidUpdate wrote: post #124

Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…

Fairly meaningless in this day and age. Also IIRC scraping legality depends heavily on jurisdiction. Some places take a more permissive view of accessing publicly available information, even if a site's TOS forbids bots. In the US there’s a major precedent [0] which held that scraping public-facing pages isn’t a CFAA "unauthorized access" issue. That’s a big part of why we’ve seen entire venture-backed scraping compa…

From that article:

> However, after further appeal in another court, hiQ was found to be in breach of LinkedIn's terms, and there was a settlement.

So why would the same not apply here?

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#154

Post by hopelite »

voidUpdate wrote: post #152
esskay wrote: post #135

Earlier quoted context omitted.

Fairly meaningless in this day and age. Also IIRC scraping legality depends heavily on jurisdiction. Some places take a more permissive view of accessing publicly available information, even if a site's TOS forbids bots. In the US there’s a major precedent [0] which held that scraping public-facing pages isn’t a CFAA "unauthorized access" issue. That’s a big part of why we’ve seen entire venture-backed scraping compa…

So if you are legally allowed to "adapt, edit or create derivative works from any materials", what's the point of the TOS?

That’s a good question. It also would not be the first time that companies use trickery and manipulation or even deliberately illegal practices for various business/financial reasons. At the very least it could be used as a tool to underpin intimidating lawsuits and another step up, regardless of the legality in the relevant jurisdiction, it could be used to influence official government foreign policy to exert pressure on a jurisdiction that permits scraping.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#155

Post by zigzag312 »

voidUpdate wrote: post #152
esskay wrote: post #135

Earlier quoted context omitted.

Fairly meaningless in this day and age. Also IIRC scraping legality depends heavily on jurisdiction. Some places take a more permissive view of accessing publicly available information, even if a site's TOS forbids bots. In the US there’s a major precedent [0] which held that scraping public-facing pages isn’t a CFAA "unauthorized access" issue. That’s a big part of why we’ve seen entire venture-backed scraping compa…

So if you are legally allowed to "adapt, edit or create derivative works from any materials", what's the point of the TOS?

I believe TOS is binding as long as it doesn't conflict with the law. If something is deemed fair use under the law, TOS cannot override those legal rights.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#158

Post by croes »

pantropy wrote: post #143
voidUpdate wrote: post #124

Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…

Technically speaking none of Goodreads material or content is being used publically, the only information displayed on the site is freely available (Title, Author) and not Goodread's property. You could try to argue that this falls under "create derivative works from any materials or content accessible on the Service" but even then it seems really flimsy to say that recommending books based on Goodread reviews is an…

I visit your garden and take 1 apple from your tree

I visit your garden and take 1000 apples from your tree.

Not that different.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#159

Post by costco »

idoubtit wrote: post #121
torben-friis wrote: post #82

Earlier quoted context omitted.

(From the site) >If you visit the "intersect" page, you can input multiple books and find the set of users that have read all of those books. This can be useful for finding longer tail books that weren't popular enough to meet the threshold. For instance, if you like reading about the collapse of the Soviet Union, you could put in "Lenin's Tomb" and "Secondhand Time", and see what other books the resultant users have…

The "Intersect" page was useless for me. I added 15 books, but got no matching user. I entered a cycle of removing-searching, and at 10 books I had 2 users: one had read 41353 books, and the other 85363, with no ratings... To be useful, the "Intersect" page should have: - find near matches when there is no exact match with every book, - ignore fake users (can any human read 80k books in many languages?), - do not ign…

What do you think the probability that someone else read 15 books you also read is? It’s very unlikely unless they are all staples of a genre, part of the same series, or just extremely popular in general. 3-5 books is how much I would use on that page. I have found interesting accounts of medievalists, people who work at think tanks, etc with it.

Fake users I would agree should be filtered, but I don’t think filtering out users who gave it a bad review is necessarily the intended behavior. If I put in 3 semi obscure Russian history books, I am presumably looking for someone who is an expert in Russian history to see what else they read. In that case I don’t care if they didn’t like one of the books or not. Approximate matches would require something like LSH or cosine similarity of average input book embedding against average embedding of read books of every user which I think wouldn’t work well anyone for retrieving anyone with a moderately long interaction history.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#160

Post by exe34 »

esskay wrote: post #135
voidUpdate wrote: post #124

Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…

Fairly meaningless in this day and age. Also IIRC scraping legality depends heavily on jurisdiction. Some places take a more permissive view of accessing publicly available information, even if a site's TOS forbids bots. In the US there’s a major precedent [0] which held that scraping public-facing pages isn’t a CFAA "unauthorized access" issue. That’s a big part of why we’ve seen entire venture-backed scraping compa…

it's only legal if you have a team of lawyers though. the law still applies to the rest of us.
Post reply on HN