Live data from Hacker News

Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

book.sv

121–130 of 275 posts

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#121
post #9

OK, I just added books until you told me I had too many. Fun idea! I have a couple of suggestions: * UI - once someone clicks "Add" you really should remove that item from the suggested list - it's very confusing to still see it. * Beam search / diversification -- Your system threw like 100 books at me of which I'd read 95 and heard of 2 of the other 3, so it worked for me as a predictor of what I'd read, but not so…

(From the site) >If you visit the "intersect" page, you can input multiple books and find the set of users that have read all of those books. This can be useful for finding longer tail books that weren't popular enough to meet the threshold. For instance, if you like reading about the collapse of the Soviet Union, you could put in "Lenin's Tomb" and "Secondhand Time", and see what other books the resultant users have…

The "Intersect" page was useless for me. I added 15 books, but got no matching user. I entered a cycle of removing-searching, and at 10 books I had 2 users: one had read 41353 books, and the other 85363, with no ratings...

To be useful, the "Intersect" page should have:

- find near matches when there is no exact match with every book,

- ignore fake users (can any human read 80k books in many languages?),

- do not ignore users' votes (my input was books I liked, I expected to find users that rated them highly).

With the "Recommend" page I had the same problem as the GP, and all the recommendations were useless. To fix that, I think some features are needed:

- do not list books by authors from my list (I don't need recommendations for them),

- add a button for marking a suggested book as "disliked" (at the bare minimum, it should remove it from the suggestion, and ideally it should influence le suggestions as much as a "liked" book),

- do not suggest several books by the same author,

- add a button to hide a suggestion or show more suggestions (there were dozens of books I'd read but wouldn't rate high).

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#122

I think this is cool and super fast -- kudos on whatever tech you needed to tackle to make it so. I don't see anyone saying safety or ethics, so I'll just put it out there that it has some safety and ethical considerations you should consider. Consider "inflammatory" books and how they could be used to harm a group of people. Although I recognize folks post this "publicly", I think the intersection feature provides m…

Thank you for the kind remarks. I would say your request is reasonable and it’s something I thought about but it’s worth noting that if you scroll down on a Goodreads book page you can see all of the users who gave it 1 star, 2 stars, etc (not just reviews, ratings too). And in fact Goodreads does not tell people this but these lists include private users. So someone looking to cause controversy would probably just find an incendiary book and look for everyone who gave it 5 stars. My data does not include people who marked their account as private or only visible to those who are signed in and also does not give information on stars. It is also very easy to remove your data if you are so inclined. I only log request URLs but I can tell that many thousands of people visited the site and less than 10 opted out despite “Remove My Data” being prominently displayed and referenced in the attached text to my post.

Also, I don’t use them but based on forum posts I read I think other services like LibraryThing or Storygraph do expose similar information about book readers.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#123
Many years ago I built a D3 graph based explorer for movies using the IMDB API. You would start by entering a movie title and it would pull up the similar movies from IMDB and you could click them and see more similar ones and they would all be connected based on similarity. It was very fun!

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#124
Does this break part 4 of the Goodreads TOS?

"[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited."

Also did the reviewers give you permission to fed their content into an LLM?

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#127

Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…

Why ask questions you already know the answers to?

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#128

Does this break part 4 of the Goodreads TOS? "[...] you agree not to sell, license, rent, modify, distribute, copy, reproduce, transmit, publicly display, publicly perform, publish, adapt, edit or create derivative works from any materials or content accessible on the Service. Use of the Goodreads Content or materials on the Service for any purpose not expressly permitted by this Agreement is strictly prohibited." Al…

I’m not taking sides in this debate, however since feeding whole books into LLMs is considered legal fair use now, I guess these reviews don’t require a permission as well. Would be great to hear a professional lawyer take on this.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#129
My impressions are similar as those of others:

- It seems to mostly show me books I've already read and know of, including sequels of what I added, which isn't very useful.

- It ultimately seems to prioritize "highest rated in category" too much, rather than focusing more on what made my chosen books stand out over others.

- Needs a "disliked books" list, especially when the recommendations show me a lot of superficially similar books I hated. I'd like to blacklist them.

- Would be cool to have a discovery mechanism for less popular and even obscure titles. Again, the top picks of each category are very well-known.

- Might not be practical, but I'd like some way to filter by specific features of reviews. E.g. prioritize reviews that say "the MC is a psychopath/murderhobo/rapist" higher for anti-recommendation, ignore reviews that say "whiny character", etc.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#130
I typed "Introduction to Real Analysis by Bartle" and I got:

Steve Jobs by Walter Isaacson

Harry Potter and the Sorcerer's Stone (Harry Potter, #1) by JK Rowling

Topology by James R Munkres

and so on.. Munkres' book is relevant and I want to read it, but what have Steve Jobs and Harry Potter got to do with with mathematics?

Post reply on HN