Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving recommendation systems and search in the age of LLMs

eugeneyan.com

Post reply on HN
Print view View original

51–60 of 98 posts

Re: Improving recommendation systems and search in the age of LLMs

#51

Post by lmeyerov »

pizza wrote: post #19
anonymousDan wrote: post #9

It's interesting that none of these papers seem to be coming out of academic labs....

Checking if a recommendation system is actually good in practice is kind of tough to do without owning a whole internet media platform as well. At best, you'll get the table scraps from these corporations (in the form of toy datasets/models made available), and you still will struggle to make your dev loop productive enough without throwing similar amounts of compute that the ~FAANGs do so as to validate whether that…

As someone whose customers do this stuff, I'm 100% for most academics chasing harder and more important problems

Most of these papers are specialized increments on high baselines for a primarily commercial problem. Likewise, they focus on optimizing phenomena that occur in their product, which may not occur in others. Eg, Netflix sliding window is neato to see the result of, but I rather students user their freedom to explore bigger ideas like mamba, and leave sliding windows to a masters student who is experimenting with intentionally narrowly scoped tweaks.

Re: Improving recommendation systems and search in the age of LLMs

#52

Post by appleorchard46 »

UrineSqueegee wrote: post #49
thaumiel wrote: post #42

ah this explains why my spotify experience has gotten worse over time.

I have the exact opposite experience, recently when a playlist I have is over, I find that every recommended track that plays after, I love so much I end up putting in my playlist

I liked when you could make a playlist radio and do that manually. That's been removed now of course.

Re: Improving recommendation systems and search in the age of LLMs

#53

Post by bookofjoe »

Perplexity Pro suggested several portable car battery chargers, which led me to search online reviews, whose consensus (five or so review sites) highest-rated chargers were the first two on Perplexity's recommendation list. In other words, the AI was an helpful guide to focused deeper search.

Re: Improving recommendation systems and search in the age of LLMs

#54

Post by lmeyerov »

anonymousDan wrote: post #9

It's interesting that none of these papers seem to be coming out of academic labs....

As someone whose customers do this stuff, I'm 100% for most academics chasing harder and more important problems.

Most of these papers are specialized increments on high baselines for a primarily commercial problem. Likewise, they focus on optimizing phenomena that occur in their product, which may not occur in others. Eg, Netflix sliding window is neato to see the result of, but I rather students user their freedom to explore bigger ideas like mamba, and leave sliding windows to a masters student who is experimenting with intentionally narrowly scoped tweaks. At that point, to top PhD grads at industrial labs will probably win.

That said, recsys is a general formulation with applications beyond shopping carts and social feeds, and bigger ideas do come out, where I'd expect competitive labs to do projects on. GNN for recsys was a big bet a couple years ago, and LLMs now, and it is curious to me those bigger shifts are industrial labs papers as you say. Maybe the statement there is recsys is one of the areas that industry hires a lot of PhDs on, as it is so core to revenue lift: academia has regular representation, while industry is overrepresented.

Re: Improving recommendation systems and search in the age of LLMs

#55

Post by memhole »

It looks like a great overview of recommendation systems. I think my main takeaways are:

1. Latency is a major issue.

2. Fine tuning can lead to major improvements and I think reduce latency. If I didn’t misread.

3. There’s some threshold or problems where prompting or fine tuning should be used.

Re: Improving recommendation systems and search in the age of LLMs

#56

Post by brookst »

amelius wrote: post #47
osmarks wrote: post #22

Earlier quoted context omitted.

You could just run a local LLM over every document and ask it "is this related to this query". I don't think you actually want to wait a week (and holding all the documents you might ever want to search would run to petabytes). (the reasonable way is embedding search, which runs much faster with some precomputation, but you still have to store things)

A better way would be to ask the LLM to generate keywords (or queries). And then use old school techniques to find a set of documents, and then filter those using another LLM.

How is that better than embeddings? You’re using embeddings to get a finite list of keywords, throwing out the extra benefits of embeddings (support for every human language, for instance), using a conventional index, and then going back to embeddings space for the final LLM?

That whole thing can be simplified to: compute and store embeddings for docs, compute embeddings for query, find most similar docs.

Re: Improving recommendation systems and search in the age of LLMs

#57

Post by amelius »

brookst wrote: post #56
amelius wrote: post #47

Earlier quoted context omitted.

A better way would be to ask the LLM to generate keywords (or queries). And then use old school techniques to find a set of documents, and then filter those using another LLM.

How is that better than embeddings? You’re using embeddings to get a finite list of keywords, throwing out the extra benefits of embeddings (support for every human language, for instance), using a conventional index, and then going back to embeddings space for the final LLM? That whole thing can be simplified to: compute and store embeddings for docs, compute embeddings for query, find most similar docs.

Yes, you can do the "old school search" part with embeddings.

Re: Improving recommendation systems and search in the age of LLMs

#58

Post by curious_cat_163 »

whatever1 wrote: post #2

Why we don’t have an LLM based search tool for our pc / smartphones? Specially for the smartphones all of your data is on the cloud anyway, instead of just scraping it for advertising and the FBI they could also do something useful for the user?

> Why we don’t have an LLM based search tool for our pc / smartphones?

I'll offer my take as an outside observer. If someone has better insights, feel free to share as well.

In market terms, I think it is because Google, Microsoft and Apple are all still trying with varied success. It has to be them because that's where a big bulk of the users are. They are all also public companies with impatient investors wanting the stock to go up into the right. So, they are both cautious about what ship to billions of devices (brand protection) and cautious about "opening up" their OS beyond that they have already done (fear of disruption).

In technical terms, it is taking a while because if the tool is going to use LLMs, then they need to solve for 99.999% of the reliability problems (brand protection) that come with that tech. They need to solve for power consumption (either on edge or in the data centers) due to their sheer scale.

So, their choices are ship fast (which Google has been trying to do more) and iterate in public; or partner with other product companies by investing in them (which Microsoft has been doing with Open AI and Google is doing with Anthropic, etc.).

Apple is taking some middle path but they just fired the person who was heading up the initiative [1] so let's see how that goes.

My two cents.

[1] https://www.reuters.com/technology/artificial-intelligence/a...

Re: Improving recommendation systems and search in the age of LLMs

#59

Post by novia »

I started listening to this article (using a text to speech model) shortly after waking up.

I thought it was very heavy on jargon. Like, it was written in a way that makes the author appear very intelligent without necessarily effectively conveying information to the audience. This is something that I've often seen authors do in academic papers, and my one published research paper (not first author) is no exception.

I'm by no means an expert in the field of ML, so perhaps I am just not the intended audience. I'm curious if other people here felt the same way when reading though.

Hopefully this observation / opinion isn't too negative.

Re: Improving recommendation systems and search in the age of LLMs

#60

Post by thaumiel »

UrineSqueegee wrote: post #49
thaumiel wrote: post #42

ah this explains why my spotify experience has gotten worse over time.

I have the exact opposite experience, recently when a playlist I have is over, I find that every recommended track that plays after, I love so much I end up putting in my playlist

My taste in music is apparently so varied, that if I want to keep the "daily" Spotify list as I want them, I have to limit myself in variation in what I listen to, otherwise they will get too mixed up and I will not enjoy them anymore. So I use other peoples recommendations or music review sites instead to find new music/bands/artists. I tried the spotify AI dj service a couple of times, but it has not been a good experience, when it tries to push in a new direction it has never really gotten it right for me.
Post reply on HN