Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

book.sv

Post reply on HN
Print view View original

101–110 of 275 posts

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#101

Post by zeroq »

Great work!

Some five years ago I was day dreaming about recommendation engine for movies where you could say "hey Ciri, give me a good gangster flick", and it will come up with something that you haven't seen yet but you'd definitely love.

To my amazement almost everyone, even true AI believers, thought it was impossible to achieve. :d

But my question is - having such huge dataset, do we really need AI for it? SASRec/RAG is sexy, but could the same result be achieved with simple ranking and intersections like lastfm did in the past with music?

Some twenty years ago I came up with an idea of "brain" data structure for recommendations where you have all your items (books, movies or articles) modeled as a graph, and whenever you pick something it makes a ripple effect, effectively raising scores in cascade of every adjecent item.

Just like your brain works - when you stumble upon something new it immediately brings back memories of similar things from the past. I never had the opportunity to implement it and test in real life scenario, but I'd be surprised if a variant of this is not widely used across different recommendation systems, like Amazon.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#102

Post by robertritz »

majormajor wrote: post #92

Neat! It's a validation of the model that 75%+ of the recommendations are things I've read and also enjoyed, with a few "read, didn't like" and some more "didn't read, don't really want to." But I think to break the content-bubble effects to find the longer tail, some way to reject or blacklist things - and have that be taken into effect in the model - might help.

To add to this Youtube afaik uses multiple models to sprinkle in new content alongside your usual recommendation just for this.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#103

Post by esafak »

zeroq wrote: post #101

Great work! Some five years ago I was day dreaming about recommendation engine for movies where you could say "hey Ciri, give me a good gangster flick", and it will come up with something that you haven't seen yet but you'd definitely love. To my amazement almost everyone, even true AI believers, thought it was impossible to achieve. :d But my question is - having such huge dataset, do we really need AI for it? SASRe…

last.fm used a primitive machine learning algorithm too, else what are you going to rank by?

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#104

Post by gen220 »

The best way I’ve found for finding predictably enjoyable fiction is to read interviews with the authors I like, and read about the works and authors they admire or are influenced by. Or who they exchanged letters or communications with, if they’re long dead and no interviews proper exist.

Strongly recommend giving that a try yourself. And trying to build an algorithm around it!

Here’s an example: Tolstoy really admired Turgenev, who was friends with Theodore Storm and Gustave Flaubert, and greatly admired Gogol.

If you like Anna Karenina you’ll probably find something of value in Torrents of Spring, Immensee, Madame Bovary or Dead Souls.

It spiders out pretty quickly!

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#105

Post by daemonologist »

majormajor wrote: post #92

Neat! It's a validation of the model that 75%+ of the recommendations are things I've read and also enjoyed, with a few "read, didn't like" and some more "didn't read, don't really want to." But I think to break the content-bubble effects to find the longer tail, some way to reject or blacklist things - and have that be taken into effect in the model - might help.

Likewise, I put in six of my favorites and had already read (and enjoyed) 29 of the 30 recommendations (I'll have to check out Blindsight by Watts). Working great but it would be cool - as with pretty much every recommendation algorithm ever - to have more of a "discovery" capability.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#107

Post by zeroq »

esafak wrote: post #103
zeroq wrote: post #101

Great work! Some five years ago I was day dreaming about recommendation engine for movies where you could say "hey Ciri, give me a good gangster flick", and it will come up with something that you haven't seen yet but you'd definitely love. To my amazement almost everyone, even true AI believers, thought it was impossible to achieve. :d But my question is - having such huge dataset, do we really need AI for it? SASRe…

last.fm used a primitive machine learning algorithm too, else what are you going to rank by?

Did they? I recall similar site back from as far as from 2008. Might be them or something similar.

Anyway. I can totally see such site running purely on statistics. Every song, every artists, every genre is a bucket. You listen to a song you put a drop in these buckets. Once there's enough water running we can compare you to other users and their buckets.

It might be hard to run it on scale in real time, but c'mon, it's leetcode junior level assignment level of complication.

Re: Show HN: I scraped 3B Goodreads reviews to train a better recommendation model

#109

Post by Semaphor »

blehn wrote: post #62

You should filter out authors from the input books in the output. If liked a book by an author, surely I'd read more of their work if I wanted to — recommending them isn't helpful. Along the same lines, I think interesting recommendations tend to be the ones that (1) I like and (2) I didn't expect. The more similar the recommendations are to the input, the more likely I already know them, and the more likely to creat…

> You should filter out authors from the input books in the output.

No, or at least make it configurable.

I’d agree for series, but not for Authors, just because I once read a book by someone doesn’t mean I even know they have other stuff, the list of Authors I read and enjoyed is very long.

Post reply on HN