Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

book.sv

Post reply on HN
Print view View original

51–60 of 275 posts

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#51

Post by dbl000 »

costco wrote: post #13
jamesponddotco wrote: post #6

The recommendations are pretty good; even though I only input six books, it was enough for it to recommend books I have on my wish list. Definitely going to play around some more. Plus, the website is super fast, very impressive. Any chance we could get an API going at some point? Are you planning to open source the work? I'm interested in the scrapping of Goodreads too. I'm building a book metadata aggregation API a…

Thank you for the compliments :) I used 50-100 datacenter proxies. I just logged requests made by the iOS app with Charles and then recreated the headers to the best of my ability though the server did not seem to be very strict at all. Worth noting though that static residential proxies are not too expensive these days anyways. Re the API: The model does actually run fairly well on CPU so it probably wouldn't be too…

I would love an API or the dataset if you could share it somehow! Just to play around with my own book lists.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#52

Post by jacquesm »

guelo wrote: post #48
__alexander wrote: post #34

Care to share the scrapped data? I would love to play around with it.

I'm surprised he got that much data. Goodreads uses several tricks to try to stop scrapers, for example pagination only works up to a few pages.

They might send him a bill for use of resources.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#53

Post by jacquesm »

qingcharles wrote: post #19

I put in a bunch of books and hit recommendations and... I'd already read 95% of them, so at least we know it works well! (checking out the other 5% now) p.s. one idea: when you click [Add] on the recommended books list, it should remove it from that list p.p.s. if there is a way to filter out the spam "Summary of ____" books, that would be good too

I have a hard time remembering titles of books I've read if they are not directly related to the subject matter. No problem remembering the content though. With movies I remember both.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#54

Post by costco »

goatsi wrote: post #27
costco wrote: post #13

Earlier quoted context omitted.

Thank you for the compliments :) I used 50-100 datacenter proxies. I just logged requests made by the iOS app with Charles and then recreated the headers to the best of my ability though the server did not seem to be very strict at all. Worth noting though that static residential proxies are not too expensive these days anyways. Re the API: The model does actually run fairly well on CPU so it probably wouldn't be too…

Speaking of sustained scraping for AI services, I found a strange file on your site: https://book.sv/robots.txt . Would you be able to explain the intent behind it?

I didn't want an agent to get stuck on an infinite loop invoking endpoints that cost GPU resources. Those fears are probably unfounded, so if people really cared I could remove those. /similar is blocked by default because I don't want 500000 "similar books for" pages to pollute the search results for my website but I do not mind if people scrape those pages.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#55

Post by xkbarkar »

Have nothing to add that hasn’t already been commented. Like the entries in the add list stay. Other than that, my recommendation list keeps coming up with books I have already read and loved and I am hitting the limit :(.

So filtering would be great,

I have seen a few versions of the same books listed more than once.

Loved this. Hope you get to tune it a little.

Also, thank you for not ruining the site with a single popup, email subscription list offer, chatbot, wheelspin from hell anywhere.

Blessings from the popup hating part of the interwebs.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#57

Post by noir_lord »

costco wrote: post #16
noir_lord wrote: post #11

It has a tendency to recommend books in the same series as are input (putting aside that if I like a book in a series I've likely already read the series). It did suggest Murderbot Diaries (not on the input but a series I have read and did like) and an Adrian Tchaikovsky I hadn't read :).

It's explicitly trained to predict the next book read in a sequence, which is why you get that behavior. There's probably a better way for me to handle it rather than having 5 books from the same series tend towards the top though.

If you have the data to know the other books in a series maybe split the results so you have "books in series" in one column and "books not in a series mentioned" in the other but other than that it did a better job than Kindle recommendations which are often hilariously off the mark.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#58

Post by costco »

dbl000 wrote: post #50

Echoing what everyone else has said here - awesome site, love how fast it was. I did notice that when I put in a single book in a series (in my case Going Postal, Discworld #33) that tended to dominate the rest of the selection. That does make sense, but I don't want recommendations for a series I'm already well into. Also noticed that a few books (Spycraft by Nadine Akkerman and Pete Langman, Tribalism is Dumb by An…

Yes I would say the handling of series is probably the biggest problem. Once my test metrics got to a point I was happy with and my quality spot checks passed (can I follow the models recommendations from one generic history book to Steven Runciman, also making sure popular books don't always dominate the results), I was ready to release because I had been working on this project for so long. The solution is probably using the transformer model to generate 100-200 candidates and then having a reranker on top.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#60

Post by teaearlgraycold »

mcbrit wrote: post #40
teaearlgraycold wrote: post #37

Earlier quoted context omitted.

I don’t think past liked books are nearly enough information to provide a good book for you today. You need a lot more information about the state of someone’s mind.

You're talking to a dude. (in my case.) I mentioned 8 books. I won't tell you exactly what to do, but one way to do it is to measure your surprise with me choosing each of those 8 books when you provide a recommendation back to me of what I should read next. I think I get kind of that experience talking to someone about books. The algorithm didn't do that.

Talking to someone about books gives you so much more information than a book list. Their expressions, their accent, their energy level, their clothes, and many other things help to provide supplemental information.
Post reply on HN